Re: Creating a Processor

[Robert Edward Herding]

Hello Patrick,

 

Thanks once more for your answer, very interesting.

I have set  my IP as you said.
Now I can access the RDF description, only if I set the IP address in the URL without namespaces, like "http://192.168.1.12:8092/sepa/org.streampipes.tutorial.geofencing"

But still doesn't import the processor

Funny thing I can access the RDF page also with the docker IP

 

Reading the logs I also found an warning, only when accessing the RDF page:

 

AM org.glassfish.jersey.server.wadl.WadlFeature configure

WARNING: JAXBContext implementation could not be found. WADL feature is disabled.

 

As you said "manage endpoints" is not connecting to Consul(inside docker), while RDF description page is(outside docker).

One question; It is possible to create an external Consul service(in the cloud) and import from there? That would be the easier/faster way ?

 

Best regards,

Robert

 

 

 

30.08.2021, 03:27, "Patrick Wiener" <wie...@apache.org>:

Hi Robert,

 

good to hear that we got it to register.

 

The second problem seems to be due to the fact that the service health check on the processor is failing given that you set the url to localhost:8092 for SP_HOST.

Thats also why you see the red dot in the overview.

 

The reason for this basically is due to the hybrid dev environment with parts of your setup running in the IDE (the processor) and the other parts inside Docker (here Consul for

service discovery and service health check). While connections from your Docker Host (your Debian machine) work on Consul over localhost:8500 (outside -> inside Docker) the

issue occurs when services like Consul try to communicate to other services running outside Docker, here on your Debian host, especially via „localhost“. The reason for this is

that Docker containers run isolated in their own namespace and are not aware of each other. While docker to docker connections work due to some convenient help from Docker 

using service DNS names (for containers within the same network), it fails for development purposes (with some exceptions). Hence, when using „localhost“ Consul thinks the service

is also running within the same container as Consul is, which is false, as it is running on your machine in your IDE without Docker.

 

So given your setup on a linux machine, what I would recommend for development is to not use „localhost“ but the actual IP of your linux machine (see ifconfig). Docker containers

can communicate to the NIC's (e.g. eth0) IP address. However, this only works on linux as Docker runs natively. Yet it is worthy to note that if you regularly change your network environments

which may lead to different IP addresses assigned to this NIC, the service health check would start failing again, as the service was originally registered with another IP.

We currently work on improving the whole service discovery/service management part to hopefully resolve all these issues within the current dev branch which is part of the next release

(0.69.0).

 

As of now, see if it helps using the machine’s IP address (e.g., 192.168.x.x) instead of localhost.

 

Best

Patrick

 

Am 29.08.2021 um 17:26 schrieb Robert Edward Herding <robertedward.herd...@yandex.com>:

 

Hello Patrick,

 

Thank you very much, your explanation helped a lot.

I was running the compose docker file, not the cli enviroment.

So I created a pipeline element template with cli, and it worked perfectly, pretty straightforward.

Now I have a registered service at Consul.

 

After that I tried to import the processor in Install Pipeline Elements, with the url given at http://localhost:8092/ (I had to change the port).

The Install Pipeline Elements shows no elements (it keeps searching) but I can see "geofence tutorial" in Pipeline Element Configuration with a red dot, for sure it is not working but it can see the container.

Can you please give me a north on this?

 

Yours sincerely,

 

Robert

 

 

 

 

 

 

28.08.2021, 10:37, "Patrick Wiener" <wie...@apache.org>:

Hi Robert,

 

first of all I’d like to welcome you to the list.

 

I assume you already started the developer environment such that Consul Docker container is running?

You can use the StreamPipes CLI to easily configure a development environment. You can find relevant parts from here [1].

 

A little bit of background: we leverage Consul for service discovery such that all extending services, e.g. processors etc

first register at Consul from where the StreamPipes core picks up service endpoints of respective extensions.

Hence, currently also for development purposes it is necessary to set of a minimal development environment using the CLI and

a corresponding environment template.

 

Can you also provide an excerpt of „docker ps -a“?

 

Cheers

Patrick

 

[1] https://streampipes.apache.org/docs/docs/extend-setup.html

 

Am 27.08.2021 um 23:19 schrieb Robert Edward Herding <robertedward.herd...@yandex.com>:

 

Hello folks,

 

I am a developer located in South Africa.

I have been testing StreamPipes for a few weeks now and I have decided to build, a processor entity for testing and developing for a future industrial use.

Said that I start following the tutorial https://streampipes.apache.org/docs/docs/extend-tutorial-data-processors.html.

The project is building perfectly, but when it gets to the point of registering items in Consul it just loops with the message.

 

22:41:53.553 SP [main] INFO  o.a.s.config.consul.ConsulSpConfig - Trying to connect to Consul to register config items

22:41:53.554 SP [main] INFO  o.a.s.config.consul.ConsulSpConfig - Retrying in 1 second

22:41:54.554 SP [main] INFO  o.a.s.config.consul.ConsulSpConfig - Trying to connect to Consul to register config items

22:41:54.555 SP [main] INFO  o.a.s.config.consul.ConsulSpConfig - Retrying in 1 second

20:41:55.555 SP [main] INFO  o.a.s.config.consul.ConsulSpConfig - Trying to connect to Consul to register config items

20:41:55.556 SP [main] INFO  o.a.s.config.consul.ConsulSpConfig - Retrying in 1 second

 

I am really not used to work with Consul, so have been really hard to understand why it can't find it.

I tried to change the SP_HOST in the run configuration.

 

localhost

0.0.0.0

127.0.0.1

192.168.1.1

192.168.1.12

host.docker.internal

 

Below the docker IP and docker network.

 

udp        0      0 192.168.1.12:bootpc     host.docker.inte:bootps ESTABLISHED
unix  3      [ ]         STREAM     CONNECTED     701942   /run/snap.docker/containerd/containerd.sock
unix  3      [ ]         STREAM     CONNECTED     692953   /run/snap.docker/containerd/containerd.sock
unix  3      [ ]         STREAM     CONNECTED     703571   /run/snap.docker/containerd/containerd.sock

 

 

NETWORK ID     NAME      DRIVER    SCOPE
305ea4177bdb   bridge    bridge    local
8c6df02eeef9   host      host      local
ead9e6d74aa1   none      null      local

 

As you can see I am running linux Debian.

Any help would be greatly appreciated.

 

Yours sincerely,

 

Robert