I was able to resolve the error. Initially I was giving custom name for
subscribe but was giving topic name at topics options.Giving the same name
at both places worked.

I am confused now with the difference of giving value for topic and
subsribe here. do you have any suggestions?

  reader.option("subscribe", topics);
reader.option("startingOffsets", startOff);
reader.option("failOnDataLoss", false);
reader.option("topic", topics);

On Wed, May 22, 2019 at 8:10 AM Gabor Somogyi <[email protected]>
wrote:

> Have you tried what the exception suggests?
>
> If startingOffsets contains specific offsets, you must specify all
> TopicPartitions.
>
> BR,
> G
>
>
> On Tue, May 21, 2019 at 9:16 PM KhajaAsmath Mohammed <
> [email protected]> wrote:
>
>> Hi,
>>
>> I am getting below errror when running sample strreaming app. does anyone
>> have resolution for this?
>>
>> JSON OFFSET {"test":{"0":0,"1":0,"2":0,"3":0,"4":0,"5":0}}
>>  ----- Herreee
>> root
>>  |-- key: string (nullable = true)
>>  |-- value: string (nullable = true)
>>  |-- topic: string (nullable = true)
>>  |-- partition: string (nullable = true)
>>  |-- offset: string (nullable = true)
>>
>> Herreee
>> No physical plan. Waiting for data.
>>
>> org.apache.spark.sql.streaming.StreamingQueryException: assertion failed:
>> If startingOffsets contains specific offsets, you must specify all
>> TopicPartitions.
>> Use -1 for latest, -2 for earliest, if you don't care.
>> Specified: Set(test-0, test-5, test-1, test-4, test-3, test-2) Assigned:
>> Set()
>> === Streaming Query ===
>> Identifier: [id = d74f6c1a-fa30-4ae3-b87d-f1b03dc6e659, runId =
>> 31e49067-8cc3-4f9c-9817-fc311ae2a417]
>> Current Committed Offsets: {}
>> Current Available Offsets: {}
>>
>> Current State: ACTIVE
>> Thread State: RUNNABLE
>>
>> Logical Plan:
>> Project [cast(key#7 as string) AS key#21, cast(value#8 as string) AS
>> value#22, cast(topic#9 as string) AS topic#23, cast(partition#10 as string)
>> AS partition#24, cast(offset#11L as string) AS offset#25]
>> +- StreamingExecutionRelation KafkaV2[Subscribe[testStream]], [key#7,
>> value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13]
>>
>>
>> java.lang.AssertionError: assertion failed: If startingOffsets contains
>> specific offsets, you must specify all TopicPartitions.
>> Use -1 for latest, -2 for earliest, if you don't care.
>> Specified: Set(test-0, test-5, test-1, test-4, test-3, test-2) Assigned:
>> Set()
>>
>> Thanks,
>> Asmath
>>
>

Reply via email to