Re: Creating a Processor

Wed, 01 Sep 2021 00:26:13 -0700

Hello Patrick,
 
Yes, it is working, you solved the problem :) !!!
Thank you very much for the last days. I have learned a lot.
 
I have to confess that this setup now doesn't look difficult at all (now), actually it is quite simple.
But working with docker and other system locally together can be a bit annoying.
 
Thanks again and lets start build components.
 
I wish all the best for StreamPipes team.
 
Best regards,
Robert Edward Hearding
 
 
 
 
31.08.2021, 10:19, "Patrick Wiener" <[email protected]>:
From your backend logs it seems that it tries to fetch element descriptions from a service referenced by the name „geofence-tutorial“. Assuming that this is the name of your processor which is running in your IDE on your machine this won’t work as „geofence-tutorial“ cannot be resolved from within Docker.
 
If you remove and delete all volumes also for Consul, and restart everythign this shouldn’t be a problem anymore, as then the processors registers with the IP address of your machine.
It can be that you initially registered it via „geofence-tutorial“. See the Config.java class
 
config.register(ConfigKeys.HOST, "geofence-tutorial", "Data processor host“);
 
It defaults to „geofence-tutorial“ if you not provide an env variable of ConfigKeys.HOST (SP_HOST). This is what you probably did the second time when starting, i.e., SP_HOST=192.168.x.x
However, upon initial startup the processor falsely registered under the address „geofence-tutorial“ for the corresponding service id which we do not overwrite on later restarts.
 
To cut the long story short, please run from clean slate and remove all containers and volumes, set SP_HOST to your IP once you start of the service from your IDE and it should work. There shouldn’t
be any references to a service under the address „geofence-tutorial“ anymore.
 
A short note: the whole service definition and discovery is completely reconstructed for the next release which will hopefully be much more easy to handle and totally get rid of these potential issues. 
 
Hope this helps :)
 
Best
Patrick
 
Am 31.08.2021 um 15:51 schrieb Robert Edward Herding <[email protected]>:
 
Hello Patrick,
 
I have attached the log file from Consul and Backend services.
 
Thanks
Robert
 
31.08.2021, 08:54, "Patrick Wiener" <[email protected]>:
Hi Robert,
 
could you please go and check the backend service logs? Maybe it provides more details on why the installation fails.
 
Patrick
 
Am 31.08.2021 um 14:49 schrieb Robert Edward Herding <[email protected]>:
 
Hi Patrick and Dominik,
 
Thanks for all the patience. I really appreciate the help.
 
Yes I am always deleting the volume to have a fresh start. But I still get the red dot.
Then I tried what Dominik said and now I have a beautiful DataProcessor Geofencing component :) .
But when I tried to install it:
 

Installing 1 of 1 (Geofencing)...error

More Details:
geofence-tutorial: System error
 
Also again I have got this warning (not sure if this is the problem, but it seems related as it only shows when its time to import the component)
 
PM org.glassfish.jersey.server.wadl.WadlFeature configure
WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
 
I will fix this warning and review the program for some mistype (just to be sure it is perfectly ok) and I hope coming back with good news.
If you guys have any more ideas please let me know.
 
 
Best regard,
Robert Edward
 
31.08.2021, 07:13, "Dominik Riemer" <[email protected]>:

Hi Robert,

on some development setups the locally started extensions service is not detected as a healthy service by Consul and thus not added to the list of pipeline elements in the UI.
You can also try to click on "Manage Endpoints" and add the URL of the extensions service manually (just add the URL such as http://192.168.1.12:8092). Maybe that's also worth a try ;-)

Dominik

On 2021/08/31 10:25:06, Patrick Wiener <[email protected]> wrote:

 Hi Robert,
 
 did you also reset everything to a clean slate using the CLI? There might be some misconfigurations due to your initial settings using „localhost“ that might not get overwritten.
 Hopefully, if you perform the following steps we get you sorted:
 
 Remove StreamPipes and perform cleanup (don’t forget the -v flag as this is required to delete the docker volumes): ./streampipes down -v
 Restart StreamPipes: ./streampipes up -d
 Finish setup process in the SP UI
 Start processor from IDE with your 192.168.x.x IP
 Check the SP UI at the „Configuration“ > „Pipeline Element Configuration“ tab and validate the green dot for your new processor service
 Go to „Install Pipeline Elements“ and check if you can install the new processor
 
 Best
 Patrick
 
 > Am 31.08.2021 um 11:18 schrieb Robert Edward Herding <[email protected]>:
 >
 > Hello Patrick,
 >
 > Thanks once more for your answer, very interesting.
 > I have set my IP as you said.
 > Now I can access the RDF description, only if I set the IP address in the URL without namespaces, like "http://192.168.1.12:8092/sepa/org.streampipes.tutorial.geofencing"
 > But still doesn't import the processor
 > Funny thing I can access the RDF page also with the docker IP
 >
 > Reading the logs I also found an warning, only when accessing the RDF page:
 >
 > AM org.glassfish.jersey.server.wadl.WadlFeature configure
 > WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
 >
 > As you said "manage endpoints" is not connecting to Consul(inside docker), while RDF description page is(outside docker).
 > One question; It is possible to create an external Consul service(in the cloud) and import from there? That would be the easier/faster way ?
 >
 > Best regards,
 > Robert
 >
 >
 >
 > 30.08.2021, 03:27, "Patrick Wiener" <[email protected]>:
 > Hi Robert,
 >
 > good to hear that we got it to register.
 >
 > The second problem seems to be due to the fact that the service health check on the processor is failing given that you set the url to localhost:8092 for SP_HOST.
 > Thats also why you see the red dot in the overview.
 >
 > The reason for this basically is due to the hybrid dev environment with parts of your setup running in the IDE (the processor) and the other parts inside Docker (here Consul for
 > service discovery and service health check). While connections from your Docker Host (your Debian machine) work on Consul over localhost:8500 (outside -> inside Docker) the
 > issue occurs when services like Consul try to communicate to other services running outside Docker, here on your Debian host, especially via „localhost“. The reason for this is
 > that Docker containers run isolated in their own namespace and are not aware of each other. While docker to docker connections work due to some convenient help from Docker
 > using service DNS names (for containers within the same network), it fails for development purposes (with some exceptions). Hence, when using „localhost“ Consul thinks the service
 > is also running within the same container as Consul is, which is false, as it is running on your machine in your IDE without Docker.
 >
 > So given your setup on a linux machine, what I would recommend for development is to not use „localhost“ but the actual IP of your linux machine (see ifconfig). Docker containers
 > can communicate to the NIC's (e.g. eth0) IP address. However, this only works on linux as Docker runs natively. Yet it is worthy to note that if you regularly change your network environments
 > which may lead to different IP addresses assigned to this NIC, the service health check would start failing again, as the service was originally registered with another IP.
 > We currently work on improving the whole service discovery/service management part to hopefully resolve all these issues within the current dev branch which is part of the next release
 > (0.69.0).
 >
 > As of now, see if it helps using the machine’s IP address (e.g., 192.168.x.x) instead of localhost.
 >
 > Best
 > Patrick
 >
 > Am 29.08.2021 um 17:26 schrieb Robert Edward Herding <[email protected] <mailto:[email protected]>>:
 >
 > Hello Patrick,
 >
 > Thank you very much, your explanation helped a lot.
 > I was running the compose docker file, not the cli enviroment.
 > So I created a pipeline element template with cli, and it worked perfectly, pretty straightforward.
 > Now I have a registered service at Consul.
 >
 > After that I tried to import the processor in Install Pipeline Elements, with the url given at http://localhost:8092/ <http://localhost:8092/> (I had to change the port).
 > The Install Pipeline Elements shows no elements (it keeps searching) but I can see "geofence tutorial" in Pipeline Element Configuration with a red dot, for sure it is not working but it can see the container.
 > Can you please give me a north on this?
 >
 > Yours sincerely,
 >
 > Robert
 >
 >
 >
 >
 >
 >
 > 28.08.2021, 10:37, "Patrick Wiener" <[email protected] <mailto:[email protected]>>:
 > Hi Robert,
 >
 > first of all I’d like to welcome you to the list.
 >
 > I assume you already started the developer environment such that Consul Docker container is running?
 > You can use the StreamPipes CLI to easily configure a development environment. You can find relevant parts from here [1].
 >
 > A little bit of background: we leverage Consul for service discovery such that all extending services, e.g. processors etc
 > first register at Consul from where the StreamPipes core picks up service endpoints of respective extensions.
 > Hence, currently also for development purposes it is necessary to set of a minimal development environment using the CLI and
 > a corresponding environment template.
 >
 > Can you also provide an excerpt of „docker ps -a“?
 >
 > Cheers
 > Patrick
 >
 > [1] https://streampipes.apache.org/docs/docs/extend-setup.html <https://streampipes.apache.org/docs/docs/extend-setup.html>
 >
 > Am 27.08.2021 um 23:19 schrieb Robert Edward Herding <[email protected] <mailto:[email protected]>>:
 >
 > Hello folks,
 >
 > I am a developer located in South Africa.
 > I have been testing StreamPipes for a few weeks now and I have decided to build, a processor entity for testing and developing for a future industrial use.
 > Said that I start following the tutorial https://streampipes.apache.org/docs/docs/extend-tutorial-data-processors.html <https://streampipes.apache.org/docs/docs/extend-tutorial-data-processors.html>.
 > The project is building perfectly, but when it gets to the point of registering items in Consul it just loops with the message.
 >
 > 22:41:53.553 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Trying to connect to Consul to register config items
 > 22:41:53.554 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Retrying in 1 second
 > 22:41:54.554 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Trying to connect to Consul to register config items
 > 22:41:54.555 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Retrying in 1 second
 > 20:41:55.555 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Trying to connect to Consul to register config items
 > 20:41:55.556 SP [main] INFO o.a.s.config.consul.ConsulSpConfig - Retrying in 1 second
 >
 > I am really not used to work with Consul, so have been really hard to understand why it can't find it.
 > I tried to change the SP_HOST in the run configuration.
 >
 > localhost
 > 0.0.0.0
 > 127.0.0.1
 > 192.168.1.1
 > 192.168.1.12
 > host.docker.internal
 >
 > Below the docker IP and docker network.
 >
 > udp 0 0 192.168.1.12:bootpc host.docker.inte:bootps ESTABLISHED
 > unix 3 [ ] STREAM CONNECTED 701942 /run/snap.docker/containerd/containerd.sock
 > unix 3 [ ] STREAM CONNECTED 692953 /run/snap.docker/containerd/containerd.sock
 > unix 3 [ ] STREAM CONNECTED 703571 /run/snap.docker/containerd/containerd.sock
 >
 >
 > NETWORK ID NAME DRIVER SCOPE
 > 305ea4177bdb bridge bridge local
 > 8c6df02eeef9 host host local
 > ead9e6d74aa1 none null local
 >
 > As you can see I am running linux Debian.
 > Any help would be greatly appreciated.
 >
 > Yours sincerely,
 >
 > Robert
 >
 
 
<backend.log><consul.log>

Reply via email to