You can put a class in the org.apache.spark namespace to access anything that is private[spark]. You can then make enrichments there to access whatever you need. Just beware upgrade pain :)
El martes, 6 de octubre de 2015, Erwan ALLAIN <eallain.po...@gmail.com> escribió: > Hello, > > I'm currently testing spark streaming with kafka. > I'm creating DirectStream with KafkaUtils and everything's fine. However I > would like to use the signature where I can specify my own message handler > (to play with partition and offset). In this case, I need to manage > offset/partition by myself to fill fromOffsets argument. > I have found a Jira on this usecase > https://issues.apache.org/jira/browse/SPARK-6714 but it has been closed > telling that it's too specific. > I'm aware that it can be done using kafka api (TopicMetaDataRequest and > OffsetRequest) but what I have to do is almost the same as the KafkaCluster > which is private. > > is it possible to : > - add another signature in KafkaUtils ? > - make KafkaCluster public ? > > or do you have any other srmart solution where I don't need to copy/paste > KafkaCluster ? > > Thanks. > > Regards, > Erwan ALLAIN >
