Hi! When trying to migrate our PDI/Spoon projects over to Apache Hop the following issue occurred, where the documentation does not give us sufficient information.
In PDI/Spoon we simple used bash-scripts to start “kitchen.sh” referencing the
main kjb-job of the ETL project (note, we are using GUI-less Linux servers).
The typical workflow is:
* We develop using Spoon (or Apache Hop) on our Windows developer machines,
and push the project to a git repos.
* We then clone (or copy) the git repos a projects into an arbitrary
directory on the Linux server and start the project with “kitchen.sh”
from there (in fact, we use a wrapper shellscript similar to the one below that
calls “kitchen.sh”).
Trying to do the same seems with Hop makes a little more problems:
* a project is needed
* a environment is needed for reading in the env.-config file(s) containing
environment variables
* so the script (see below, is stored as part of every project with
different variable values) that calls “hop-run.sh” now also uses “hop-conf.sh”
* to (delete and) create an environment
* to (delete and) create a temporary project in the current file
location that is deleted again when the ETL is finished
* We find this procedure clumsy and is probably not the recommended
procedure for such a use case.
NOTE: We do not want to have to pre-define/pre-create all potential Hop
projects for all potential file locations on the Linux server, as we are having
many projects, and want to keep the flexibility to run them inside arbitrary
file locations on the server.
#!/bin/bash
WORKFLOW_TO_START=test_workflow.hwf
# removes the last 4 characters:
HOP_HWF=${WORKFLOW_TO_START%????}
CURRENT_FOLDER=$(pwd)
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
CURRENT_LOGFILE=${HOP_HWF}_${TIMESTAMP}.log
HOP_PATH=/home/pentaho/hop/apache-hop-client-2.3.0/hop
PROJECT_PATH=$(pwd)
PROJECT_NAME=$(basename $(pwd))
ENV_NAME=test-env
ENV_PURPOSE=Development
ENV_CONF_FILE=env-var.json
RUN_CONFIG=local
DEBUG_LEVEL=BASIC
sh ${HOP_PATH}/hop-conf.sh --project-delete --project ${PROJECT_NAME} >
/dev/null 2>&1
sh ${HOP_PATH}/hop-conf.sh --project-create --project ${PROJECT_NAME}
--project-home=${PROJECT_PATH}
sh ${HOP_PATH}/hop-conf.sh --environment-delete --environment ${ENV_NAME} >
/dev/null 2>&1
sh ${HOP_PATH}/hop-conf.sh --environment-create --environment ${ENV_NAME}
--environment-project ${PROJECT_NAME} --environment-purpose=${ENV_PURPOSE}
--environment-config-files=${PROJECT_PATH}/${ENV_CONF_FILE}
sh ${HOP_PATH}/hop-run.sh --project ${PROJECT_NAME} --file ${WORKFLOW_TO_START}
--runconfig=${RUN_CONFIG} --environment=${ENV_NAME} --level=${DEBUG_LEVEL} >>
$CURRENT_LOGFILE 2>&1 &
sh ${HOP_PATH}/hop-conf.sh --project-delete --project ${PROJECT_NAME} >
/dev/null 2>&1
mkdir -p $CURRENT_FOLDER/logs
mv $CURRENT_LOGFILE logs
We cannot imagine that this is the way we should continue. What would be the
recommended way?
Side note:
Our first approach was to use the already existing “samples” or “default”
project on the server, and simply re-direct its ${PROJECT_HOME} to the file
location where ETL-project we currently want to start is stored.
So we tried to manipulate the variable using “project-modify” switch:
sh /home/pentaho/hop/apache-hop-client-2.3.0/hop/hop-conf.sh --project-modify
--project default --project-home=/home/pentaho/hop/testhoprun/
This indeed gives the message that the project-home variable would have been
updated successfully:
Project configuration for 'default' was modified in
/home/pentaho/hop/apache-hop-client-2.3.0/hop/config/hop-config.json
BUT looking at hop-config.json we see that nothing has changed, the
project-home variable is still pointing to the original location of the default
project.
(And the workflow stored in the testhoprun location cannot be executed
succesfully, of course.)
This is independent of trying to manipulate ${PROJECT_HOME} of the “default”,
“sample” or any other existing project.
Is this a bug?
(Tested on Hop 2.3).
Mit freundlichen Grüßen
Gerhard Mitterlechner
Senior Development Engineer
Database Management
[email protected]<mailto:[email protected]>
________________________________
[eurofunk Kappacher Logo]
eurofunk KAPPACHER GmbH
eurofunk-Straße 1 − 8, 5600 St. Johann im Pongau, Österreich / Austria
Board of Management: Christian Kappacher, Dr. Christian Kappacher
VAT No.: ATU35454003 | Registered in: FN 52582 b, LG Salzburg
________________________________
Products | Solutions | Service
www.eurofunk.com<http://www.eurofunk.com/>
[https://servicedesk.eurofunk.com/helplinePortal/content/images/eurofunk_facebook.png]<https://www.facebook.com/eurofunkKappacher/>
[https://servicedesk.eurofunk.com/helplinePortal/content/images/eurofunk_linkedin.png]
<https://www.linkedin.com/company/5039801/>
[https://servicedesk.eurofunk.com/helplinePortal/content/images/eurofunk_youtube.png]
<https://www.youtube.com/user/eurofunkkappacher/>
[https://servicedesk.eurofunk.com/helplinePortal/content/images/eurofunk_xing.png]
<https://www.xing.com/companies/eurofunkkappachergmbh>
smime.p7s
Description: S/MIME Cryptographic Signature
