Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/15102
> I'd want to see some test cases though that show why the current
implementation is wrong from an end-user perspective if it needs to block
merging initial kafka support.
PR with failing test indicating at least one reason why it's wrong from an
end-user perspective:
https://github.com/zsxwing/spark/pull/4
> I do not think it is reasonable to suggest we block merging this patch
on an overhaul of the DataSource API configuration system.
Here's what I actually said:
'if you know your plan down the line is to use json for structured
configuration, you should use it now, and provide more convenient ways to
construct json later, not use "convenient" non-json hacks now.'
No hyperbole about blocking on a complete overhaul, nothing that isn't
backwards compatible. I'm just saying that, if the design document already
recognizes that json is necessary to work around the string -> string
interface... start using structured json strings now, and make it more
convenient later.
Or do you actually think that stuff like
option("assign", "topicA:1:1,topicA:2:2,topicB:3:3")
makes it clear what the arguments are?
> I think @koeninger made a good suggestion to block accepting certain
kafka configurations.
In case it wasn't clear, I was not suggesting that preventing users from
doing things they could otherwise do with Kafka is actually a good idea. I
think it's a bad idea, but if you're going to run with it, you might as well be
consistent about it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
