Spark-avro 4.0.0 is released

The 4.0.0 release adds support for Spark 2.2. The published artifact is compatible with both Spark 2.1 and 2.2. New Features: - Support for Spark 2.2 (#242 ): resolve compatibility issue with datasource write API changes

Re: spark-stream memory table global?

It must be accessed under the same SparkSession. We can also add an option to make it be a global temp view. Feel free to open a PR to improve it. On Fri, Nov 10, 2017 at 4:56 AM, Imran Rajjad wrote: > Hi, > > Does the memory table in which spark-structured streaming results

Re: Generate windows on processing time in Spark Structured Streaming

Hmmm, we should allow that. current_timestamp() is acutally deterministic within any given batch. Could you open a JIRA ticket? On Fri, Nov 10, 2017 at 1:52 AM, wangsan wrote: > Hi all, > > How can I use current processing time to generate windows in streaming > processing? >

Parquet files from spark not readable in Cascading

Hi, When I tried reading parquet data that was generated by spark in cascading it throws following error Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file "" at

Spark Streaming Kafka

Hi All, I’m new to streaming avro records and am parsing Avro from a Kafka direct stream with spark streaming 2.1.1, I was wondering if anyone could please suggest an API for decoding Avro records with Scala? I’ve found KafkaAvroDecoder, twitter/bijection and the Avro library, each seem to

spark-stream memory table global?

Hi, Does the memory table in which spark-structured streaming results are sinked into, is available to other spark applications on the cluster? Is it by default global or will only be available to context where streaming is being done thanks Imran -- I.R

Generate windows on processing time in Spark Structured Streaming

Hi all, How can I use current processing time to generate windows in streaming processing? window function's Scala doc says "For a streaming query, you may use the function current_timestamp to generate windows on processing time.” But when using current_timestamp as column in window