Re: [I] Allow Manual Upload of Sample Event for Adapter Schema Inference
tenthe closed issue #4139: Allow Manual Upload of Sample Event for Adapter Schema Inference URL: https://github.com/apache/streampipes/issues/4139 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Allow Manual Upload of Sample Event for Adapter Schema Inference
tenthe commented on issue #4139: URL: https://github.com/apache/streampipes/issues/4139#issuecomment-3825739956 Thanks a lot for the detailed description of your problem. The `AdapterException` during the adapter update or startup typically indicates that the adapter could not be started successfully on the worker after the schema change. Since a restart resolves the issue temporarily, this could be related either to an inconsistent adapter state or to a schema incompatibility introduced during the update process. To better understand and reproduce the issue, could you please share: 1. **A sample JSON event before the schema change** 2. **A sample JSON event after the schema change** Additionally, if you are using a **preprocessing or transformation script**, please share that as well, as it may influence how the schema is interpreted or validated during adapter startup. With these details, it should be much easier to reproduce the issue. Also, feel free to ask any questions you may have regarding StreamPipes concepts in the GitHub discussions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Allow Manual Upload of Sample Event for Adapter Schema Inference
intigration commented on issue #4139: URL: https://github.com/apache/streampipes/issues/4139#issuecomment-3824998039 I recently started working with Apache StreamPipes and am still in the early phase of understanding the platform. also, I am using a SNAPSHOT version which could surface such glitches, As per yours recommendation. I attempted to update the adapter with a new/updated schema. After applying the change, the update returned an error. I restarted the services and everything looks normal https://github.com/user-attachments/assets/df036179-37e3-4171-b41a-d6a687e99d13"; /> for the interest, here are the error logs org.apache.streampipes.commons.exceptions.connect.AdapterException at org.apache.streampipes.connect.management.management.WorkerRestClient.triggerAdapterStateChange(WorkerRestClient.java:111) at org.apache.streampipes.connect.management.management.WorkerRestClient.startAdapter(WorkerRestClient.java:89) at org.apache.streampipes.connect.management.management.WorkerRestClient.invokeStreamAdapter(WorkerRestClient.java:60) at org.apache.streampipes.connect.management.management.AdapterMasterManagement.startStreamAdapter(AdapterMasterManagement.java:185) at org.apache.streampipes.connect.management.management.AdapterUpdateManagement.updateAdapter(AdapterUpdateManagement.java:107) at org.apache.streampipes.rest.impl.connect.AdapterResource.updateAdapter(AdapterResource.java:117) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:67) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:727) at org.apache.streampipes.rest.impl.connect.AdapterResource$$SpringCGLIB$$0.updateAdapter() at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:257) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:190) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:986) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:891) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1088) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:978) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:925) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:593) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
Re: [I] Allow Manual Upload of Sample Event for Adapter Schema Inference
intigration commented on issue #4139: URL: https://github.com/apache/streampipes/issues/4139#issuecomment-3824946450 @tenthe Thanks for your analysis. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Allow Manual Upload of Sample Event for Adapter Schema Inference
tenthe commented on issue #4139:
URL: https://github.com/apache/streampipes/issues/4139#issuecomment-3824458753
Hi @intigration,
thanks for reporting this.
If I understand you correctly, you are trying to publish **two different
kinds of events (with different event schemas) to the same topic**.
This is **not directly supported in StreamPipes**. A StreamPipes adapter
assumes that all incoming events conform to **one single event schema**.
While it is possible to have *incomplete events* (i.e., some fields may be
missing), it is **not possible to handle two completely different schemas
within the same adapter**.
### Examples of different event schemas on the same topic
**Example 1 – Temperature event**
```json
{
"deviceId": "sensor-1",
"timestamp": 17,
"temperature": 21.5
}
```
**Example 2 – Status event**
```json
{
"deviceId": "sensor-1",
"timestamp": 170005,
"status": "offline"
}
```
Although both events might be sent to the same topic, they represent
**different schemas** and therefore cannot be handled by a single adapter.
If the schema of the events on a topic changes over time, the adapter must
be **updated accordingly**.
If you have **multiple different event schemas on one topic**, you can use
the **new preprocessing functions** (currently available only in the **SNAPSHOT
version**, but they will be part of the next release):
* Create **one adapter per event schema**
* Use preprocessing functions to **filter out events that do not match the
expected schema**
This way, each adapter only processes the events it is designed for.
Here’s a simple and clear example you can use, plus a short description that
fits nicely into your previous text.
**Sample JavaScript filter function (status field)**
This function forwards only those events that contain a `status` field.
Events without this field are ignored.
```javascript
// Forwards only events that contain a "status" field
function transform(event, out, ctx) {
if (event.status !== undefined && event.status !== null) {
// returns the same event
out.collect(event);
}
}
```
Alternatively, if there are many different event types on one topic, you can
leverage the **compact adapter API**, which allows you to avoid creating a
large number of adapters manually via the UI.
I hope this helps, let us know if you have further questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] Allow Manual Upload of Sample Event for Adapter Schema Inference
intigration commented on issue #4139: URL: https://github.com/apache/streampipes/issues/4139#issuecomment-3823033220 I also face a kind of problem with schema handling of broker based data sources , Initially I configured the mqtt adapter to subscribe my datasource, as you the schema in attached snapshot. later I use the same topic subscription to push the differents events. you can see only status and time from new event data is updating - https://github.com/user-attachments/assets/ea24c6cb-580c-4484-97a8-5699d021158d"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
