advancedxy commented on code in PR #166: URL: https://github.com/apache/arrow-datafusion-comet/pull/166#discussion_r1512141931
########## .github/actions/setup-spark-builder/action.yaml: ########## @@ -0,0 +1,64 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: Setup Spark Builder +description: 'Setup Apache Spark to run SQL tests' +inputs: + spark-short-version: + description: 'The Apache Spark short version (e.g., 3.4) to build' + required: true + default: '3.4' + spark-version: + description: 'The Apache Spark version (e.g., 3.4.2) to build' + required: true + default: '3.4.2' + comet-version: + description: 'The Comet version to use for Spark' + required: true + default: '0.1.0-SNAPSHOT' +runs: + using: "composite" + steps: + - name: Clone Spark repo + uses: actions/checkout@v4 + with: + repository: apache/spark + path: apache-spark + ref: v${{inputs.spark-version}} + fetch-depth: 1 + + - name: Setup Spark for Comet + shell: bash + run: | + cd apache-spark + git apply ../dev/diffs/${{inputs.spark-version}}.diff Review Comment: I see. > On the other hand, the diff is tied to a particular Spark version like 3.4.x, and rarely need to be updated (from our experience) But once modification is needed, it would be problematic to update the patches directly. If we are going to go with this approach. I'd like to propose some improvement to refine the maintaining process: 1. It would be ideal that we can host all these patches in a dedicated branch(s) per spark release in a forked spark repo. The repo should be public and ideally is hosted under one organization. 2. It might not be appropriate to reference the forked branch directly in an apache project nor I can think of a repo that can be served for this purpose. Therefore, I think maybe we can host the dedicated branch in your personal spark repo: `sunchao/spark` to start? 3. Once modifications are needed. The modifications should go to the forked branches first. When modification is merged, it would be pretty straightforward to generate the patch using `git` command. 4. Submit a PR with the dedicated patch in this repo. By hosting patches in a dedicated branch, I think we can track all the modifications in history. Of course, we should include a README in dev/diffs about how the diffs are generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
