Hi,
first of all THANK YOU FOR APACHE HOP!
For all of the following questions
- I haven't found information in the documentation so please direct me to if allready available.
- using client/UI env Windows Server 2012 R2
- using execution env Docker container based on doc setup on CentOs 7
1) It would be nice to have some kind of indented infrastructure/environment when using Apache Hop
why: It is not clear to me how you indented to execute Hop Worklflows.
Something like
Something like
- UI only
Run Apache Hop GUI and execute with the default local run configuration
Run Apache Hop GUI and execute with the default local run configuration
- UI and Hop server
Run Apache Hop GUI, run Hop serveradd a remote execution run configuration and use it as run configuartion, execute workflows via UI
In my point of view missing and somehow mentioned also in the videos are scenarios where the dev and execution process is splitted meaning execution is not trigger via the UI but via some external trigger like a cron job
- UI + Docker as execution platform
develop in Hop GUI, copy/push project files into a file repository like GIT
?? CI/CD to rebuild the docker container with the current project files ??
-> as Hop Run is changing project files in the project folder we have to provide a pure copy of the project files if GIT is used it won't be possible to pull changes based on local changes of files !!! (this seems to me more like a archetecture failure: why is hop run changing project files? may switch to use a working directory for Hop Run)
- UI + Hop Server as execution platform
as the server is started with a set of project files which will not be updated throughout its lifetime we would have to restart the server everytime changes occure
-> is there a grace kill waiting for any running workflow/pipe to finish before the server is killed
-> will the server loose execution information on restarts
-> there is currently no API documentation for Hop Server which makes the useage somekind of hard
advanced scenarios and how would they look like
- UI + Hop Server > Hop Server
- UI + Hop Server & Beam
...
conclusion: it would be nice to have a section of possible Hop infrastructure examples for development and execution as a blueprint or starting point maybe with some kind of picture of the infrastructure and which components are needed and how they are related to each other
possible bug:
if an environment is derived from another and both are available in the docker file image the base env metadatafiles are not used/copied automatically but must be added manually to the metadata of the configured execution project
How should the project be setup if you want to use inheritance?
2) indented usage of GIT and the hop-config.json
as by now hop-config.json is defining the relationship between a project and an environment
"lifecycleEnvironments" : [ {
"name" : "prod_docker_exec",
"purpose" : "Production",
"projectName" : "its_dv_etl",
"configurationFiles" : [ "${PROJECT_HOME}/environment\\prod_docker_exec-config.json" ]
}
"lifecycleEnvironments" : [ {
"name" : "prod_docker_exec",
"purpose" : "Production",
"projectName" : "its_dv_etl",
"configurationFiles" : [ "${PROJECT_HOME}/environment\\prod_docker_exec-config.json" ]
}
This would mean that hop-config.json must be part of the project files and thatfor part of the GIT repository. In conclusion you may only have one GIT repository and thatfor "project" per hop folder.
How are multi GIT projects are intended to work? Under which folder should we place our GIT repository in that case?
-> why is a project related configuration done in an ui config file? may move env config from hop-config.json to project-config.json
possible bug: the UI is not filtering the environments based on the selected project but switching projects based on the selected environment
3) trivial tests and current node state
I'm sorry if these things are allready known but you may know a workaround and/or if unknown by now may add a bug in Jira
# the script nodes do not seem to support all internal defined types,
examples are
- Java node returning Decimal where even a BigDecimal type is not registered at Rhino
- _javascript_ node can't convert numbers at all (from my last test)
-> may set up a policy that each node should have a trivial test verifiying that all supported types can be used as input and can be generated and used as output
# not working in ui by now in my env
Execute SQL script does not allow to add Parameters, current workaround Java node and Execute row SQL
-> is this intended as newer frameworks like spring would support to add all the fields making the use of dedicated Parameters obsolete. If so how can we use the paramters? Please add an example to the doc.
-> is this intended as newer frameworks like spring would support to add all the fields making the use of dedicated Parameters obsolete. If so how can we use the paramters? Please add an example to the doc.
- it's not possile to set the database connection by a env variable, as a workaround we can set the database connection fields by variables but this seems more like a bug then indented
# possible improvements
if we are working with env variables (multi env like win/linux) the Workflow/Pipline target path should support some kind of easy use, by now we have to choose a file and replace the leading part with the variable placeholder on all nodes referencing a file like A Pipeline or Workflow nodes
if we are working with env variables (multi env like win/linux) the Workflow/Pipline target path should support some kind of easy use, by now we have to choose a file and replace the leading part with the variable placeholder on all nodes referencing a file like A Pipeline or Workflow nodes
may add the possibility to have a multiline window for the Java _expression_ node even if single line (short statements) are indented. By now the field is resizing to its content and linebreaks are not allowed making more complex statements some kind of hard to use.
4) documentation
some of the tables in the doc are wronly build as field names and descriptions are mixed and occuring on both sides of the table. see Pipeline > Parameter as example
some UI links to doc are incorrect like with the Http client node directing to Get variables
Thank you and sorry for the long mail.
Br,
Peter
