Hristo Angelov created SPARK-23232: -------------------------------------- Summary: Mapping Dataset to a Java bean always set 1L to a long field Key: SPARK-23232 URL: https://issues.apache.org/jira/browse/SPARK-23232 Project: Spark Issue Type: Bug Components: Java API Affects Versions: 2.2.1, 2.2.0 Reporter: Hristo Angelov
I have the following streaming query: {code:java} baseDataSet .groupBy(window(col(UTC_DATE_TIME), applicationProperties.getProperty("current_active_users_window_length") + " minutes", "5 seconds")) .agg(approx_count_distinct(col(INTERNAL_USER_ID), applicationProperties.getDoubleProperty("approximate_distinct_count_error_percentage")).as("value")) .filter(col("window.end").leq(current_timestamp())) .select(unix_timestamp(col("window.end")).as("timestamp"), col("value")) .writeStream() .trigger(Trigger.ProcessingTime(applicationProperties.getIntegerProperty("current_active_users_trigger_interval"), TimeUnit.SECONDS)) .format(ActiveUsersSinkProvider.class.getCanonicalName()) .outputMode(OutputMode.Update()) .option("checkpointLocation", SystemProperties.APP_CHECKPOINT_DIR + "/current_active_users") .start();{code} In the sink I'm trying to map the dataset to a Java bean with the following code: {code:java} data.as(Encoders.bean(LongTimeBased.class)).collectAsList() {code} where LongTimeBased is: {code:java} public class LongTimeBased { private long timestamp; private long value; public long getTimestamp() { return timestamp; } public void setTimestamp(long timestamp) { this.timestamp = timestamp; } public long getValue() { return value; } public void setValue(long value) { this.value = value; } } {code} So whatever data is aggregated value field is always 1. When I select the value field from every row, its value is correct: {code:java} for(Row row : data.collectAsList()) { Long value = row.getAs("value"); //correct value; } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org