Hi Gerhard, Projects and environments were introduced because of the lack of flexibility when working with multiple projects, in multiple environments. Many people and organizations started using their own scripts to circumvent those limitations and juggling with multiple kettle.properties files that were copied, parsed/populated from templates, etc. Apache Hop aims to provide a more unified and flexible way to deal with that.
Your initial approach, modifying the project home for the default project, revealed a bug[1], (thanks for reporting it). Once that's fixed, that could be a possible approach. Maybe not the best one, but at least it should work. An alternative approach that could make your life a lot easier is to run your workflows and pipelines through docker[2]. Running a pipeline or workflow in the short-lived container would come down to more or less a hybrid version of the two approaches you describe: you could mount _any_ folder on your server's file system as the project folder, and add the necessary environment files. All of the hop-conf and hop-run commands will be taken care of by the container. If you really, really, really don't want to use projects and environments, you can just remove the projects plugin from hop/plugins/misc/projects. You'll need to find another way to provide environment-specific configurations in that case, through variables, parameters, system properties ... Let us know if any of these approaches work for you or if there's anything else we can do to help. [1] https://github.com/apache/hop/issues/2494 [2] https://hop.apache.org/tech-manual/latest/docker-container.html#_how_to_run_the_container Regards, Bart On Wed, Mar 1, 2023 at 8:57 AM MITTERLECHNER Gerhard < [email protected]> wrote: > Hi! > > > > When trying to migrate our PDI/Spoon projects over to Apache Hop the > following issue occurred, where the documentation does not give us > sufficient information. > > > > In PDI/Spoon we simple used bash-scripts to start “kitchen.sh” referencing > the main kjb-job of the ETL project (note, we are using GUI-less Linux > servers). > > > > The typical workflow is: > > - We develop using Spoon (or Apache Hop) on our Windows developer > machines, and push the project to a git repos. > - We then clone (or copy) the git repos a projects into an arbitrary > directory on the Linux server and start the project with “kitchen.sh” > > from there (in fact, we use a wrapper shellscript similar to the one below > that calls “kitchen.sh”). > > > > Trying to do the same seems with Hop makes a little more problems: > > - a project is needed > - a environment is needed for reading in the env.-config file(s) > containing environment variables > - so the script (see below, is stored as part of every project with > different variable values) that calls “hop-run.sh” now also uses > “hop-conf.sh” > - to (delete and) create an environment > - to (delete and) create a temporary project in the current file > location that is deleted again when the ETL is finished > - We find this procedure clumsy and is probably not the recommended > procedure for such a use case. > > > > NOTE: We do not want to have to pre-define/pre-create all potential Hop > projects for all potential file locations on the Linux server, as we are > having many projects, and want to keep the flexibility to run them inside > arbitrary file locations on the server. > > > > #!/bin/bash > > WORKFLOW_TO_START=test_workflow.hwf > > # removes the last 4 characters: > > HOP_HWF=${WORKFLOW_TO_START%????} > > > > CURRENT_FOLDER=$(pwd) > > TIMESTAMP=$(date +"%Y%m%d_%H%M%S") > > CURRENT_LOGFILE=${HOP_HWF}_${TIMESTAMP}.log > > > > HOP_PATH=/home/pentaho/hop/apache-hop-client-2.3.0/hop > > PROJECT_PATH=$(pwd) > > PROJECT_NAME=$(basename $(pwd)) > > ENV_NAME=test-env > > ENV_PURPOSE=Development > > ENV_CONF_FILE=env-var.json > > RUN_CONFIG=local > > DEBUG_LEVEL=BASIC > > > > sh ${HOP_PATH}/hop-conf.sh --project-delete --project ${PROJECT_NAME} > > /dev/null 2>&1 > > sh ${HOP_PATH}/hop-conf.sh --project-create --project ${PROJECT_NAME} > --project-home=${PROJECT_PATH} > > > > sh ${HOP_PATH}/hop-conf.sh --environment-delete --environment ${ENV_NAME} > > /dev/null 2>&1 > > sh ${HOP_PATH}/hop-conf.sh --environment-create --environment ${ENV_NAME} > --environment-project ${PROJECT_NAME} --environment-purpose=${ENV_PURPOSE} > --environment-config-files=${PROJECT_PATH}/${ENV_CONF_FILE} > > > > sh ${HOP_PATH}/hop-run.sh --project ${PROJECT_NAME} --file > ${WORKFLOW_TO_START} --runconfig=${RUN_CONFIG} --environment=${ENV_NAME} > --level=${DEBUG_LEVEL} >> $CURRENT_LOGFILE 2>&1 & > > > > sh ${HOP_PATH}/hop-conf.sh --project-delete --project ${PROJECT_NAME} > > /dev/null 2>&1 > > > > mkdir -p $CURRENT_FOLDER/logs > > mv $CURRENT_LOGFILE logs > > > > > > We cannot imagine that this is the way we should continue. What would be > the recommended way? > > > > > > > > Side note: > > Our *first approach* was to use the already existing “samples” or > “default” project on the server, and simply re-direct its ${PROJECT_HOME} > to the file location where ETL-project we currently want to start is stored. > > So we tried to manipulate the variable using “project-modify” switch: > > sh /home/pentaho/hop/apache-hop-client-2.3.0/hop/hop-conf.sh > --project-modify --project default > --project-home=/home/pentaho/hop/testhoprun/ > > This indeed gives the message that the project-home variable would have > been updated successfully: > > Project configuration for 'default' was modified in > /home/pentaho/hop/apache-hop-client-2.3.0/hop/config/hop-config.json > > BUT looking at hop-config.json we see that *nothing has changed, *the > project-home variable is still pointing to the original location of the > default project. > > (And the workflow stored in the testhoprun location cannot be executed > succesfully, of course.) > > > > This is independent of trying to manipulate ${PROJECT_HOME} of the > “default”, “sample” or any other existing project. > > > > Is this a bug? > > (Tested on Hop 2.3). > > > > > > Mit freundlichen Grüßen > > *Gerhard Mitterlechner* > Senior Development Engineer > Database Management > > > [email protected] > ------------------------------ > > > [image: eurofunk Kappacher Logo] > > eurofunk KAPPACHER GmbH > eurofunk-Straße 1 − 8, 5600 St. Johann im Pongau, Österreich / Austria > > Board of Management: Christian Kappacher, Dr. Christian Kappacher > VAT No.: ATU35454003 | Registered in: FN 52582 b, LG Salzburg > ------------------------------ > > Products | Solutions | Service www.eurofunk.com > > <https://www.facebook.com/eurofunkKappacher/> > <https://www.linkedin.com/company/5039801/> > <https://www.youtube.com/user/eurofunkkappacher/> > <https://www.xing.com/companies/eurofunkkappachergmbh> > > > > >
