[
https://issues.apache.org/jira/browse/BEAM-8941?focusedWorklogId=374478&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-374478
]
ASF GitHub Bot logged work on BEAM-8941:
----------------------------------------
Author: ASF GitHub Bot
Created on: 20/Jan/20 09:45
Start Date: 20/Jan/20 09:45
Worklog Time Spent: 10m
Work Description: kkucharc commented on pull request #10543: [BEAM-8941]
Implement simple DSL for load tests
URL: https://github.com/apache/beam/pull/10543#discussion_r368425453
##########
File path: .test-infra/jenkins/LoadTestConfig.groovy
##########
@@ -0,0 +1,621 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonTestProperties.Runner
+import CommonTestProperties.SDK
+import LoadTestConfig.SerializableOption
+import groovy.json.JsonBuilder
+import org.codehaus.groovy.runtime.InvokerHelper
+
+import java.util.function.Predicate
+
+import static java.util.Objects.nonNull
+import static java.util.Objects.requireNonNull
+
+/**
+ * This class contains simple DSL for load tests configuration. Configuration
as Map<String, Serializable>
+ * [{@link LoadTestConfig#config config} -- returns configuration map]
+ * [{@link LoadTestConfig#templateConfig templateConfig} -- return
LoadTestConfig reusable object]
+ * [{@link LoadTestConfig#fromTemplate fromTemplate} -- returns configuration
from given template].<br><br>
+ *
+ * Example:
+ * <blockquote><pre>
+ * LoadTestConfig template = templateConfig {
+ * title 'Load test'
+ * test 'Beam Load test'
+ * dataflow()
+ * pipelineOptions {
+ * python()
+ * jobName 'Any job name'
+ * publishToBigQuery true
+ * //other fields
+ * }
+ * specificParameters([
+ * fanout: 4
+ * ])
+ * }
+ * Map<String, Serializable> configMap = fromTemplate(template) {
+ * //fields can be changed or/and added
+ * flink()
+ * pipelineOptions {
+ * parallelism 5
+ * inputOptions {
+ * numRecords 20000
+ * keySize 1000
+ * valueSize 10
+ * }
+ * }
+ * }
+ * </pre></blockquote>
+ */
+class LoadTestConfig implements SerializableOption<Map<String, Serializable>> {
+
+ private String _title
+ private String _test
+ private Runner _runner
+ private PipelineOptions _pipelineOptions
+
+ private LoadTestConfig() {}
+
+ void title(final String title) { _title = title }
+ void test(final String test) { _test = test }
+
+ //runners
+ void dataflow() { setRunnerAndUpdatePipelineOptions(Runner.DATAFLOW)}
+ void spark() { setRunnerAndUpdatePipelineOptions(Runner.SPARK) }
Review comment:
Do you have any load tests in Spark or Flink (non-portable)? I am asking
because those two options can be added later. My main concern is here if we are
sure if someone will actually run test with Spark/Flink option if it won't fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 374478)
Time Spent: 2h 10m (was: 2h)
> Create a common place for Load Tests configuration
> --------------------------------------------------
>
> Key: BEAM-8941
> URL: https://issues.apache.org/jira/browse/BEAM-8941
> Project: Beam
> Issue Type: Sub-task
> Components: testing
> Reporter: Kamil Wasilewski
> Assignee: Pawel Pasterz
> Priority: Minor
> Fix For: Not applicable
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The Apache Beam community maintains different versions of each Load Test. For
> example, right now, there are two versions of all Python Load Tests: the
> first one runs on Dataflow runner, and the second one runs on Flink. With the
> lack of a common place where configuration for the tests can be stored, the
> configuration is duplicated many times with minimal differences.
> The goal is to create a common place for the configuration, so that it could
> be passed to different files with tests (.test-infra/jenkins/*.groovy) and
> filtered according to needs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
