Re: Monitoring system extensibility

2016-10-10 Thread Pete Robbins 
Yes I agree. I'm not sure how important this is anyway. It's a little
annoying but easy to work around.

On Mon, 10 Oct 2016 at 09:01 Reynold Xin  wrote:

> I just took a quick look and set a target version on the JIRA. But Pete I
> think the primary problem with the JIRA and pull request is that it really
> just argues (or implements) opening up a private API, which is a valid
> point but there are a lot more that needs to be done before making some
> private API public.
>
> At the very least, we need to answer the following:
>
> 1. Is the existing API maintainable? E.g. Is it OK to just expose coda
> hale metrics in the API? Do we need to worry about dependency conflicts?
> Should we wrap it?
>
> 2. Is the existing API sufficiently general (to cover use cases)? What
> about security related setup?
>
>
>
>
> On Fri, Oct 7, 2016 at 2:03 AM, Pete Robbins  wrote:
>
> Which has happened. The last comment being in August with someone saying
> it was important to them. They PR has been around since March and despite a
> request to be reviewed has not got any committer's attention. Without that,
> it is going nowhere. The historic Jiras requesting other sinks such as
> Kafka, OpenTSBD etc have also been ignored.
>
> So for now we continue creating classes in o.a.s package.
>
> On Fri, 7 Oct 2016 at 09:50 Reynold Xin  wrote:
>
> So to be constructive and in order to actually open up these APIs, it
> would be useful for users to comment on the JIRA ticket on their use cases
> (rather than "I want this to be public"), and then we can design an API
> that would address those use cases. In some cases the solution is to just
> make the existing internal API public. But turning some internal API public
> without thinking about whether those APIs are sufficiently expressive and
> maintainable is not a great way to design APIs in general.
>
> On Friday, October 7, 2016, Pete Robbins  wrote:
>
> I brought this up last year and there was a Jira raised:
> https://issues.apache.org/jira/browse/SPARK-14151
>
> For now I just have my SInk and Source in an o.a.s package name which is
> not ideal but the only way round this.
>
> On Fri, 7 Oct 2016 at 08:30 Reynold Xin  wrote:
>
> They have always been private, haven't they?
>
>
> https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/metrics/source/Source.scala
>
>
>
> On Thu, Oct 6, 2016 at 7:38 AM, Alexander Oleynikov  > wrote:
>
> Hi.
>
> As of v2.0.1, the traits `org.apache.spark.metrics.source.Source` and
> `org.apache.spark.metrics.sink.Sink` are defined as private to ‘spark’
> package, so it becomes troublesome to create a new implementation in the
> user’s code (but still possible in a hacky way).
> This seems to be the only missing piece to allow extension of the metrics
> system and I wonder whether is was conscious design decision to limit the
> visibility. Is it possible to broaden the visibility scope for these traits
> in the future versions?
>
> Thanks,
> Alexander
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
>
>

Re: Monitoring system extensibility

2016-10-10 Thread Reynold Xin 
I just took a quick look and set a target version on the JIRA. But Pete I
think the primary problem with the JIRA and pull request is that it really
just argues (or implements) opening up a private API, which is a valid
point but there are a lot more that needs to be done before making some
private API public.

At the very least, we need to answer the following:

1. Is the existing API maintainable? E.g. Is it OK to just expose coda hale
metrics in the API? Do we need to worry about dependency conflicts? Should
we wrap it?

2. Is the existing API sufficiently general (to cover use cases)? What
about security related setup?




On Fri, Oct 7, 2016 at 2:03 AM, Pete Robbins  wrote:

> Which has happened. The last comment being in August with someone saying
> it was important to them. They PR has been around since March and despite a
> request to be reviewed has not got any committer's attention. Without that,
> it is going nowhere. The historic Jiras requesting other sinks such as
> Kafka, OpenTSBD etc have also been ignored.
>
> So for now we continue creating classes in o.a.s package.
>
> On Fri, 7 Oct 2016 at 09:50 Reynold Xin  wrote:
>
>> So to be constructive and in order to actually open up these APIs, it
>> would be useful for users to comment on the JIRA ticket on their use cases
>> (rather than "I want this to be public"), and then we can design an API
>> that would address those use cases. In some cases the solution is to just
>> make the existing internal API public. But turning some internal API public
>> without thinking about whether those APIs are sufficiently expressive and
>> maintainable is not a great way to design APIs in general.
>>
>> On Friday, October 7, 2016, Pete Robbins  wrote:
>>
>>> I brought this up last year and there was a Jira raised:
>>> https://issues.apache.org/jira/browse/SPARK-14151
>>>
>>> For now I just have my SInk and Source in an o.a.s package name which is
>>> not ideal but the only way round this.
>>>
>>> On Fri, 7 Oct 2016 at 08:30 Reynold Xin  wrote:
>>>
 They have always been private, haven't they?

 https://github.com/apache/spark/blob/branch-1.6/core/
 src/main/scala/org/apache/spark/metrics/source/Source.scala



 On Thu, Oct 6, 2016 at 7:38 AM, Alexander Oleynikov <
 oleyniko...@gmail.com> wrote:

> Hi.
>
> As of v2.0.1, the traits `org.apache.spark.metrics.source.Source` and
> `org.apache.spark.metrics.sink.Sink` are defined as private to
> ‘spark’ package, so it becomes troublesome to create a new implementation
> in the user’s code (but still possible in a hacky way).
> This seems to be the only missing piece to allow extension of the
> metrics system and I wonder whether is was conscious design decision to
> limit the visibility. Is it possible to broaden the visibility scope for
> these traits in the future versions?
>
> Thanks,
> Alexander
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: Monitoring system extensibility

2016-10-07 Thread Pete Robbins 
Which has happened. The last comment being in August with someone saying it
was important to them. They PR has been around since March and despite a
request to be reviewed has not got any committer's attention. Without that,
it is going nowhere. The historic Jiras requesting other sinks such as
Kafka, OpenTSBD etc have also been ignored.

So for now we continue creating classes in o.a.s package.

On Fri, 7 Oct 2016 at 09:50 Reynold Xin  wrote:

> So to be constructive and in order to actually open up these APIs, it
> would be useful for users to comment on the JIRA ticket on their use cases
> (rather than "I want this to be public"), and then we can design an API
> that would address those use cases. In some cases the solution is to just
> make the existing internal API public. But turning some internal API public
> without thinking about whether those APIs are sufficiently expressive and
> maintainable is not a great way to design APIs in general.
>
> On Friday, October 7, 2016, Pete Robbins  wrote:
>
> I brought this up last year and there was a Jira raised:
> https://issues.apache.org/jira/browse/SPARK-14151
>
> For now I just have my SInk and Source in an o.a.s package name which is
> not ideal but the only way round this.
>
> On Fri, 7 Oct 2016 at 08:30 Reynold Xin  wrote:
>
> They have always been private, haven't they?
>
>
> https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/metrics/source/Source.scala
>
>
>
> On Thu, Oct 6, 2016 at 7:38 AM, Alexander Oleynikov  > wrote:
>
> Hi.
>
> As of v2.0.1, the traits `org.apache.spark.metrics.source.Source` and
> `org.apache.spark.metrics.sink.Sink` are defined as private to ‘spark’
> package, so it becomes troublesome to create a new implementation in the
> user’s code (but still possible in a hacky way).
> This seems to be the only missing piece to allow extension of the metrics
> system and I wonder whether is was conscious design decision to limit the
> visibility. Is it possible to broaden the visibility scope for these traits
> in the future versions?
>
> Thanks,
> Alexander
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
>

Re: Monitoring system extensibility

2016-10-07 Thread Reynold Xin 
So to be constructive and in order to actually open up these APIs, it would
be useful for users to comment on the JIRA ticket on their use cases
(rather than "I want this to be public"), and then we can design an API
that would address those use cases. In some cases the solution is to just
make the existing internal API public. But turning some internal API public
without thinking about whether those APIs are sufficiently expressive and
maintainable is not a great way to design APIs in general.

On Friday, October 7, 2016, Pete Robbins  wrote:

> I brought this up last year and there was a Jira raised:
> https://issues.apache.org/jira/browse/SPARK-14151
>
> For now I just have my SInk and Source in an o.a.s package name which is
> not ideal but the only way round this.
>
> On Fri, 7 Oct 2016 at 08:30 Reynold Xin  > wrote:
>
>> They have always been private, haven't they?
>>
>> https://github.com/apache/spark/blob/branch-1.6/core/
>> src/main/scala/org/apache/spark/metrics/source/Source.scala
>>
>>
>>
>> On Thu, Oct 6, 2016 at 7:38 AM, Alexander Oleynikov <
>> oleyniko...@gmail.com
>> > wrote:
>>
>>> Hi.
>>>
>>> As of v2.0.1, the traits `org.apache.spark.metrics.source.Source` and
>>> `org.apache.spark.metrics.sink.Sink` are defined as private to ‘spark’
>>> package, so it becomes troublesome to create a new implementation in the
>>> user’s code (but still possible in a hacky way).
>>> This seems to be the only missing piece to allow extension of the
>>> metrics system and I wonder whether is was conscious design decision to
>>> limit the visibility. Is it possible to broaden the visibility scope for
>>> these traits in the future versions?
>>>
>>> Thanks,
>>> Alexander
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>> 
>>>
>>>
>>

Re: Monitoring system extensibility

2016-10-07 Thread Pete Robbins 
I brought this up last year and there was a Jira raised:
https://issues.apache.org/jira/browse/SPARK-14151

For now I just have my SInk and Source in an o.a.s package name which is
not ideal but the only way round this.

On Fri, 7 Oct 2016 at 08:30 Reynold Xin  wrote:

> They have always been private, haven't they?
>
>
> https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/metrics/source/Source.scala
>
>
>
> On Thu, Oct 6, 2016 at 7:38 AM, Alexander Oleynikov  > wrote:
>
> Hi.
>
> As of v2.0.1, the traits `org.apache.spark.metrics.source.Source` and
> `org.apache.spark.metrics.sink.Sink` are defined as private to ‘spark’
> package, so it becomes troublesome to create a new implementation in the
> user’s code (but still possible in a hacky way).
> This seems to be the only missing piece to allow extension of the metrics
> system and I wonder whether is was conscious design decision to limit the
> visibility. Is it possible to broaden the visibility scope for these traits
> in the future versions?
>
> Thanks,
> Alexander
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
>

Re: Monitoring system extensibility

2016-10-07 Thread Reynold Xin 
They have always been private, haven't they?

https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/metrics/source/Source.scala



On Thu, Oct 6, 2016 at 7:38 AM, Alexander Oleynikov 
wrote:

> Hi.
>
> As of v2.0.1, the traits `org.apache.spark.metrics.source.Source` and
> `org.apache.spark.metrics.sink.Sink` are defined as private to ‘spark’
> package, so it becomes troublesome to create a new implementation in the
> user’s code (but still possible in a hacky way).
> This seems to be the only missing piece to allow extension of the metrics
> system and I wonder whether is was conscious design decision to limit the
> visibility. Is it possible to broaden the visibility scope for these traits
> in the future versions?
>
> Thanks,
> Alexander
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Monitoring system extensibility

2016-10-06 Thread Alexander Oleynikov 
Hi. 

As of v2.0.1, the traits `org.apache.spark.metrics.source.Source` and 
`org.apache.spark.metrics.sink.Sink` are defined as private to ‘spark’ package, 
so it becomes troublesome to create a new implementation in the user’s code 
(but still possible in a hacky way).
This seems to be the only missing piece to allow extension of the metrics 
system and I wonder whether is was conscious design decision to limit the 
visibility. Is it possible to broaden the visibility scope for these traits in 
the future versions?

Thanks,
Alexander
-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org