This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new c352993 CAMEL-16577 add support to set the index time (#5504)
c352993 is described below
commit c3529935424230450654ed8fbe8815e42c5460a3
Author: iridian <442359+iridian...@users.noreply.github.com>
AuthorDate: Fri May 14 23:42:12 2021 -0700
CAMEL-16577 add support to set the index time (#5504)
* CAMEL-16577 add support to set the index time
* CAMEL-16577 added header overrides and examples for overriding the index
time
* CAMEL-16577 simplifying type conversions
Co-authored-by: Dillon Gilmore <dgil...@rei.com>
---
.../camel/catalog/docs/splunk-hec-component.adoc | 44 +++++++++++++++++++++-
.../splunkhec/SplunkHECEndpointConfigurer.java | 3 ++
.../splunkhec/SplunkHECEndpointUriFactory.java | 3 +-
.../camel/component/splunkhec/splunk-hec.json | 1 +
.../src/main/docs/splunk-hec-component.adoc | 44 +++++++++++++++++++++-
.../component/splunkhec/SplunkHECComponent.java | 1 +
.../splunkhec/SplunkHECConfiguration.java | 13 +++++++
.../component/splunkhec/SplunkHECProducer.java | 17 ++++++++-
.../splunkhec/SplunkHECConfigurationTest.java | 6 +++
.../dsl/SplunkHECEndpointBuilderFactory.java | 31 +++++++++++++++
.../modules/ROOT/pages/splunk-hec-component.adoc | 44 +++++++++++++++++++++-
11 files changed, 201 insertions(+), 6 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/splunk-hec-component.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/splunk-hec-component.adoc
index b3f66e7..93e6bd4 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/splunk-hec-component.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/splunk-hec-component.adoc
@@ -70,7 +70,7 @@ with the following path and query parameters:
|===
-=== Query Parameters (9 parameters):
+=== Query Parameters (10 parameters):
[width="100%",cols="2,5,^1,2",options="header"]
@@ -83,11 +83,24 @@ with the following path and query parameters:
| *lazyStartProducer* (producer) | Whether the producer should be started lazy
(on the first message). By starting lazy you can use this to allow CamelContext
and routes to startup in situations where a producer may otherwise fail during
starting and cause the route to fail being started. By deferring this startup
to be lazy then the startup failure can be handled during routing messages via
Camel's routing error handlers. Beware that when the first message is processed
then creating and [...]
| *source* (producer) | Splunk source argument | camel | String
| *sourceType* (producer) | Splunk sourcetype argument | camel | String
+| *time* (producer) | Time this even occurred. By default, the time will be
when this event hits the splunk server. | | Long
| *https* (security) | Contact HEC over https. | true | boolean
| *skipTlsVerify* (security) | Splunk HEC TLS verification. | false | boolean
|===
// endpoint options: END
+== Message Headers
+
+[width="100%",cols="10%,20%,70%",options="header",]
+|=======================================================================
+|Name |Type |Description
+
+|`SplunkComponent.INDEX_TIME` |`Long` | Epoch-formatted time.
+Specify with the time query string parameter. Sets a default for all events in
the request.
+The default time can be overridden.
+
+|=======================================================================
+
== Message body
The body must be serializable to JSON so it may be sent to Splunk.
@@ -98,4 +111,33 @@ The Splunk HEC endpoint may be used to stream data to
Splunk for ingestion.
It is meant for high volume ingestion of machine data.
+== Configuring the index time
+
+By default, the index time for an event is when the event makes it to the
Splunk
+server.
+
+[source,java]
+-------------------------------
+from("direct:start")
+ .to("splunk-hec://localhost:8080?token");
+-------------------------------
+
+If you are ingesting a large enough dataset with a big enough lag then the time
+the event hits the server and when that event actually happened could be
skewed. If
+you want to override the index time you can do so.
+
+[source,java]
+-------------------------------
+from("kafka:logs")
+ .setHeader(SplunkHECComponent.INDEX_TIME,
simple("${headers[kafka.HEADERS].lastKey('TIME')}"))
+ .to("splunk-hec://localhost:8080/token");
+-------------------------------
+
+[source,java]
+-------------------------------
+from("kafka:logs")
+
.toD("splunk-hec://localhost:8080/token?time=${headers[kafka.HEADERS].lastKey('TIME')}");
+-------------------------------
+
+
include::{page-component-version}@camel-spring-boot::page$splunk-starter.adoc[]
diff --git
a/components/camel-splunk-hec/src/generated/java/org/apache/camel/component/splunkhec/SplunkHECEndpointConfigurer.java
b/components/camel-splunk-hec/src/generated/java/org/apache/camel/component/splunkhec/SplunkHECEndpointConfigurer.java
index a29b7f4..24bfee9 100644
---
a/components/camel-splunk-hec/src/generated/java/org/apache/camel/component/splunkhec/SplunkHECEndpointConfigurer.java
+++
b/components/camel-splunk-hec/src/generated/java/org/apache/camel/component/splunkhec/SplunkHECEndpointConfigurer.java
@@ -35,6 +35,7 @@ public class SplunkHECEndpointConfigurer extends
PropertyConfigurerSupport imple
case "source":
target.getConfiguration().setSource(property(camelContext,
java.lang.String.class, value)); return true;
case "sourcetype":
case "sourceType":
target.getConfiguration().setSourceType(property(camelContext,
java.lang.String.class, value)); return true;
+ case "time": target.getConfiguration().setTime(property(camelContext,
java.lang.Long.class, value)); return true;
default: return false;
}
}
@@ -56,6 +57,7 @@ public class SplunkHECEndpointConfigurer extends
PropertyConfigurerSupport imple
case "source": return java.lang.String.class;
case "sourcetype":
case "sourceType": return java.lang.String.class;
+ case "time": return java.lang.Long.class;
default: return null;
}
}
@@ -78,6 +80,7 @@ public class SplunkHECEndpointConfigurer extends
PropertyConfigurerSupport imple
case "source": return target.getConfiguration().getSource();
case "sourcetype":
case "sourceType": return target.getConfiguration().getSourceType();
+ case "time": return target.getConfiguration().getTime();
default: return null;
}
}
diff --git
a/components/camel-splunk-hec/src/generated/java/org/apache/camel/component/splunkhec/SplunkHECEndpointUriFactory.java
b/components/camel-splunk-hec/src/generated/java/org/apache/camel/component/splunkhec/SplunkHECEndpointUriFactory.java
index 90dc43f..4cfd23c 100644
---
a/components/camel-splunk-hec/src/generated/java/org/apache/camel/component/splunkhec/SplunkHECEndpointUriFactory.java
+++
b/components/camel-splunk-hec/src/generated/java/org/apache/camel/component/splunkhec/SplunkHECEndpointUriFactory.java
@@ -20,7 +20,7 @@ public class SplunkHECEndpointUriFactory extends
org.apache.camel.support.compon
private static final Set<String> PROPERTY_NAMES;
private static final Set<String> SECRET_PROPERTY_NAMES;
static {
- Set<String> props = new HashSet<>(11);
+ Set<String> props = new HashSet<>(12);
props.add("lazyStartProducer");
props.add("bodyOnly");
props.add("splunkURL");
@@ -29,6 +29,7 @@ public class SplunkHECEndpointUriFactory extends
org.apache.camel.support.compon
props.add("host");
props.add("index");
props.add("source");
+ props.add("time");
props.add("https");
props.add("token");
props.add("headersOnly");
diff --git
a/components/camel-splunk-hec/src/generated/resources/org/apache/camel/component/splunkhec/splunk-hec.json
b/components/camel-splunk-hec/src/generated/resources/org/apache/camel/component/splunkhec/splunk-hec.json
index c4c0d7a..651a369 100644
---
a/components/camel-splunk-hec/src/generated/resources/org/apache/camel/component/splunkhec/splunk-hec.json
+++
b/components/camel-splunk-hec/src/generated/resources/org/apache/camel/component/splunkhec/splunk-hec.json
@@ -35,6 +35,7 @@
"lazyStartProducer": { "kind": "parameter", "displayName": "Lazy Start
Producer", "group": "producer", "label": "producer", "required": false, "type":
"boolean", "javaType": "boolean", "deprecated": false, "autowired": false,
"secret": false, "defaultValue": false, "description": "Whether the producer
should be started lazy (on the first message). By starting lazy you can use
this to allow CamelContext and routes to startup in situations where a producer
may otherwise fail during sta [...]
"source": { "kind": "parameter", "displayName": "Source", "group":
"producer", "label": "", "required": false, "type": "string", "javaType":
"java.lang.String", "deprecated": false, "autowired": false, "secret": false,
"defaultValue": "camel", "configurationClass":
"org.apache.camel.component.splunkhec.SplunkHECConfiguration",
"configurationField": "configuration", "description": "Splunk source argument"
},
"sourceType": { "kind": "parameter", "displayName": "Source Type",
"group": "producer", "label": "", "required": false, "type": "string",
"javaType": "java.lang.String", "deprecated": false, "autowired": false,
"secret": false, "defaultValue": "camel", "configurationClass":
"org.apache.camel.component.splunkhec.SplunkHECConfiguration",
"configurationField": "configuration", "description": "Splunk sourcetype
argument" },
+ "time": { "kind": "parameter", "displayName": "Time", "group": "producer",
"label": "", "required": false, "type": "integer", "javaType":
"java.lang.Long", "deprecated": false, "autowired": false, "secret": false,
"configurationClass":
"org.apache.camel.component.splunkhec.SplunkHECConfiguration",
"configurationField": "configuration", "description": "Time this even occurred.
By default, the time will be when this event hits the splunk server." },
"https": { "kind": "parameter", "displayName": "Https", "group":
"security", "label": "security", "required": false, "type": "boolean",
"javaType": "boolean", "deprecated": false, "autowired": false, "secret":
false, "defaultValue": true, "configurationClass":
"org.apache.camel.component.splunkhec.SplunkHECConfiguration",
"configurationField": "configuration", "description": "Contact HEC over https."
},
"skipTlsVerify": { "kind": "parameter", "displayName": "Skip Tls Verify",
"group": "security", "label": "security", "required": false, "type": "boolean",
"javaType": "boolean", "deprecated": false, "autowired": false, "secret":
false, "defaultValue": false, "configurationClass":
"org.apache.camel.component.splunkhec.SplunkHECConfiguration",
"configurationField": "configuration", "description": "Splunk HEC TLS
verification." }
}
diff --git
a/components/camel-splunk-hec/src/main/docs/splunk-hec-component.adoc
b/components/camel-splunk-hec/src/main/docs/splunk-hec-component.adoc
index b3f66e7..93e6bd4 100644
--- a/components/camel-splunk-hec/src/main/docs/splunk-hec-component.adoc
+++ b/components/camel-splunk-hec/src/main/docs/splunk-hec-component.adoc
@@ -70,7 +70,7 @@ with the following path and query parameters:
|===
-=== Query Parameters (9 parameters):
+=== Query Parameters (10 parameters):
[width="100%",cols="2,5,^1,2",options="header"]
@@ -83,11 +83,24 @@ with the following path and query parameters:
| *lazyStartProducer* (producer) | Whether the producer should be started lazy
(on the first message). By starting lazy you can use this to allow CamelContext
and routes to startup in situations where a producer may otherwise fail during
starting and cause the route to fail being started. By deferring this startup
to be lazy then the startup failure can be handled during routing messages via
Camel's routing error handlers. Beware that when the first message is processed
then creating and [...]
| *source* (producer) | Splunk source argument | camel | String
| *sourceType* (producer) | Splunk sourcetype argument | camel | String
+| *time* (producer) | Time this even occurred. By default, the time will be
when this event hits the splunk server. | | Long
| *https* (security) | Contact HEC over https. | true | boolean
| *skipTlsVerify* (security) | Splunk HEC TLS verification. | false | boolean
|===
// endpoint options: END
+== Message Headers
+
+[width="100%",cols="10%,20%,70%",options="header",]
+|=======================================================================
+|Name |Type |Description
+
+|`SplunkComponent.INDEX_TIME` |`Long` | Epoch-formatted time.
+Specify with the time query string parameter. Sets a default for all events in
the request.
+The default time can be overridden.
+
+|=======================================================================
+
== Message body
The body must be serializable to JSON so it may be sent to Splunk.
@@ -98,4 +111,33 @@ The Splunk HEC endpoint may be used to stream data to
Splunk for ingestion.
It is meant for high volume ingestion of machine data.
+== Configuring the index time
+
+By default, the index time for an event is when the event makes it to the
Splunk
+server.
+
+[source,java]
+-------------------------------
+from("direct:start")
+ .to("splunk-hec://localhost:8080?token");
+-------------------------------
+
+If you are ingesting a large enough dataset with a big enough lag then the time
+the event hits the server and when that event actually happened could be
skewed. If
+you want to override the index time you can do so.
+
+[source,java]
+-------------------------------
+from("kafka:logs")
+ .setHeader(SplunkHECComponent.INDEX_TIME,
simple("${headers[kafka.HEADERS].lastKey('TIME')}"))
+ .to("splunk-hec://localhost:8080/token");
+-------------------------------
+
+[source,java]
+-------------------------------
+from("kafka:logs")
+
.toD("splunk-hec://localhost:8080/token?time=${headers[kafka.HEADERS].lastKey('TIME')}");
+-------------------------------
+
+
include::{page-component-version}@camel-spring-boot::page$splunk-starter.adoc[]
diff --git
a/components/camel-splunk-hec/src/main/java/org/apache/camel/component/splunkhec/SplunkHECComponent.java
b/components/camel-splunk-hec/src/main/java/org/apache/camel/component/splunkhec/SplunkHECComponent.java
index c6a2da5..fbf27b5 100644
---
a/components/camel-splunk-hec/src/main/java/org/apache/camel/component/splunkhec/SplunkHECComponent.java
+++
b/components/camel-splunk-hec/src/main/java/org/apache/camel/component/splunkhec/SplunkHECComponent.java
@@ -24,6 +24,7 @@ import org.apache.camel.support.DefaultComponent;
@Component("splunk-hec")
public class SplunkHECComponent extends DefaultComponent {
+ public static final String INDEX_TIME = "SplunkHECIndexTime";
public SplunkHECComponent() {
}
diff --git
a/components/camel-splunk-hec/src/main/java/org/apache/camel/component/splunkhec/SplunkHECConfiguration.java
b/components/camel-splunk-hec/src/main/java/org/apache/camel/component/splunkhec/SplunkHECConfiguration.java
index 558224f..5a06bf7 100644
---
a/components/camel-splunk-hec/src/main/java/org/apache/camel/component/splunkhec/SplunkHECConfiguration.java
+++
b/components/camel-splunk-hec/src/main/java/org/apache/camel/component/splunkhec/SplunkHECConfiguration.java
@@ -44,6 +44,8 @@ public class SplunkHECConfiguration {
private boolean bodyOnly;
@UriParam
private boolean headersOnly;
+ @UriParam
+ private Long time;
public String getSourceType() {
return sourceType;
@@ -140,4 +142,15 @@ public class SplunkHECConfiguration {
public void setHeadersOnly(boolean headersOnly) {
this.headersOnly = headersOnly;
}
+
+ /**
+ * Time this even occurred. By default, the time will be when this event
hits the splunk server.
+ */
+ public Long getTime() {
+ return time;
+ }
+
+ public void setTime(Long time) {
+ this.time = time;
+ }
}
diff --git
a/components/camel-splunk-hec/src/main/java/org/apache/camel/component/splunkhec/SplunkHECProducer.java
b/components/camel-splunk-hec/src/main/java/org/apache/camel/component/splunkhec/SplunkHECProducer.java
index 3d5f88a..d2ca106 100644
---
a/components/camel-splunk-hec/src/main/java/org/apache/camel/component/splunkhec/SplunkHECProducer.java
+++
b/components/camel-splunk-hec/src/main/java/org/apache/camel/component/splunkhec/SplunkHECProducer.java
@@ -35,11 +35,14 @@ import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.ssl.SSLContextBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
* The Splunk HEC producer.
*/
public class SplunkHECProducer extends DefaultProducer {
+ private static final Logger LOG =
LoggerFactory.getLogger(SplunkHECProducer.class);
private static final ObjectMapper MAPPER = new ObjectMapper();
private SplunkHECEndpoint endpoint;
private CloseableHttpClient httpClient;
@@ -99,7 +102,7 @@ public class SplunkHECProducer extends DefaultProducer {
Map<String, Object> createPayload(Message message) {
Object body = message.getBody();
Map<String, Object> payload = new HashMap<>();
- buildPayload(payload);
+ buildPayload(message, payload);
if (endpoint.getConfiguration().isBodyOnly()) {
payload.put("event", body);
@@ -115,7 +118,7 @@ public class SplunkHECProducer extends DefaultProducer {
return payload;
}
- private void buildPayload(Map<String, Object> payload) {
+ private void buildPayload(Message message, Map<String, Object> payload) {
if (endpoint.getConfiguration().getSourceType() != null) {
payload.put("sourcetype",
endpoint.getConfiguration().getSourceType());
}
@@ -128,5 +131,15 @@ public class SplunkHECProducer extends DefaultProducer {
if (endpoint.getConfiguration().getHost() != null) {
payload.put("host", endpoint.getConfiguration().getHost());
}
+
+ Long time = message.getHeader(
+ SplunkHECComponent.INDEX_TIME,
+ endpoint.getConfiguration().getTime(),
+ Long.class
+ );
+
+ if (time != null) {
+ payload.put("time", time);
+ }
}
}
diff --git
a/components/camel-splunk-hec/src/test/java/org/apache/camel/component/splunkhec/SplunkHECConfigurationTest.java
b/components/camel-splunk-hec/src/test/java/org/apache/camel/component/splunkhec/SplunkHECConfigurationTest.java
index a3fa183..c377994 100644
---
a/components/camel-splunk-hec/src/test/java/org/apache/camel/component/splunkhec/SplunkHECConfigurationTest.java
+++
b/components/camel-splunk-hec/src/test/java/org/apache/camel/component/splunkhec/SplunkHECConfigurationTest.java
@@ -77,4 +77,10 @@ public class SplunkHECConfigurationTest {
SplunkHECConfiguration config = new SplunkHECConfiguration();
assertEquals(false, config.isHeadersOnly());
}
+
+ @Test
+ public void testDefaultTime() {
+ SplunkHECConfiguration config = new SplunkHECConfiguration();
+ assertEquals(null, config.getTime());
+ }
}
diff --git
a/core/camel-endpointdsl/src/generated/java/org/apache/camel/builder/endpoint/dsl/SplunkHECEndpointBuilderFactory.java
b/core/camel-endpointdsl/src/generated/java/org/apache/camel/builder/endpoint/dsl/SplunkHECEndpointBuilderFactory.java
index 95c5f30..92fb38d 100644
---
a/core/camel-endpointdsl/src/generated/java/org/apache/camel/builder/endpoint/dsl/SplunkHECEndpointBuilderFactory.java
+++
b/core/camel-endpointdsl/src/generated/java/org/apache/camel/builder/endpoint/dsl/SplunkHECEndpointBuilderFactory.java
@@ -206,6 +206,37 @@ public interface SplunkHECEndpointBuilderFactory {
return this;
}
/**
+ * Time this even occurred. By default, the time will be when this
event
+ * hits the splunk server.
+ *
+ * The option is a: <code>java.lang.Long</code> type.
+ *
+ * Group: producer
+ *
+ * @param time the value to set
+ * @return the dsl builder
+ */
+ default SplunkHECEndpointBuilder time(Long time) {
+ doSetProperty("time", time);
+ return this;
+ }
+ /**
+ * Time this even occurred. By default, the time will be when this
event
+ * hits the splunk server.
+ *
+ * The option will be converted to a
+ * <code>java.lang.Long</code> type.
+ *
+ * Group: producer
+ *
+ * @param time the value to set
+ * @return the dsl builder
+ */
+ default SplunkHECEndpointBuilder time(String time) {
+ doSetProperty("time", time);
+ return this;
+ }
+ /**
* Contact HEC over https.
*
* The option is a: <code>boolean</code> type.
diff --git a/docs/components/modules/ROOT/pages/splunk-hec-component.adoc
b/docs/components/modules/ROOT/pages/splunk-hec-component.adoc
index 2a63552..ba19984 100644
--- a/docs/components/modules/ROOT/pages/splunk-hec-component.adoc
+++ b/docs/components/modules/ROOT/pages/splunk-hec-component.adoc
@@ -72,7 +72,7 @@ with the following path and query parameters:
|===
-=== Query Parameters (9 parameters):
+=== Query Parameters (10 parameters):
[width="100%",cols="2,5,^1,2",options="header"]
@@ -85,11 +85,24 @@ with the following path and query parameters:
| *lazyStartProducer* (producer) | Whether the producer should be started lazy
(on the first message). By starting lazy you can use this to allow CamelContext
and routes to startup in situations where a producer may otherwise fail during
starting and cause the route to fail being started. By deferring this startup
to be lazy then the startup failure can be handled during routing messages via
Camel's routing error handlers. Beware that when the first message is processed
then creating and [...]
| *source* (producer) | Splunk source argument | camel | String
| *sourceType* (producer) | Splunk sourcetype argument | camel | String
+| *time* (producer) | Time this even occurred. By default, the time will be
when this event hits the splunk server. | | Long
| *https* (security) | Contact HEC over https. | true | boolean
| *skipTlsVerify* (security) | Splunk HEC TLS verification. | false | boolean
|===
// endpoint options: END
+== Message Headers
+
+[width="100%",cols="10%,20%,70%",options="header",]
+|=======================================================================
+|Name |Type |Description
+
+|`SplunkComponent.INDEX_TIME` |`Long` | Epoch-formatted time.
+Specify with the time query string parameter. Sets a default for all events in
the request.
+The default time can be overridden.
+
+|=======================================================================
+
== Message body
The body must be serializable to JSON so it may be sent to Splunk.
@@ -100,4 +113,33 @@ The Splunk HEC endpoint may be used to stream data to
Splunk for ingestion.
It is meant for high volume ingestion of machine data.
+== Configuring the index time
+
+By default, the index time for an event is when the event makes it to the
Splunk
+server.
+
+[source,java]
+-------------------------------
+from("direct:start")
+ .to("splunk-hec://localhost:8080?token");
+-------------------------------
+
+If you are ingesting a large enough dataset with a big enough lag then the time
+the event hits the server and when that event actually happened could be
skewed. If
+you want to override the index time you can do so.
+
+[source,java]
+-------------------------------
+from("kafka:logs")
+ .setHeader(SplunkHECComponent.INDEX_TIME,
simple("${headers[kafka.HEADERS].lastKey('TIME')}"))
+ .to("splunk-hec://localhost:8080/token");
+-------------------------------
+
+[source,java]
+-------------------------------
+from("kafka:logs")
+
.toD("splunk-hec://localhost:8080/token?time=${headers[kafka.HEADERS].lastKey('TIME')}");
+-------------------------------
+
+
include::{page-component-version}@camel-spring-boot::page$splunk-starter.adoc[]
