I am not sure it is good idea to send requests to 3rd-party addresses from Ignite node. Let's do not make the same mistakes again.
On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov <anovi...@gridgain.com> wrote: > We may directly send request to GA from Ignite node: > https://developers.google.com/analytics/devguides/collection/protocol/v1/ > <https://developers.google.com/analytics/devguides/collection/protocol/v1/ > > > Latest version can be received from maven central: > https://repo1.maven.org/maven2/org/apache/ignite/ > ignite-core/maven-metadata.xml <https://repo1.maven.org/ > maven2/org/apache/ignite/ignite-core/maven-metadata.xml> > > > > 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <dsetrak...@apache.org> > написал(а): > > > > Denis, > > > > I am not sure I understand. We already do have CGI enabled for > > download.cgi. Is there something else we need? > > > > D. > > > > On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dma...@gridgain.com> wrote: > > > >> There is an obstacle. There is no way to execute the script using PHP or > >> similar sever side language and trigger GA as discussed earlier: > >> https://issues.apache.org/jira/browse/INFRA-15182 > >> > >> How else can we tackle this? > >> > >> Denis > >> > >> On Thursday, September 7, 2017, Dmitriy Setrakyan < > dsetrak...@apache.org> > >> wrote: > >> > >>> I think it is safe to assume at this point that everyone is in general > >>> agreement, since there are no active objections. > >>> > >>> I have filed a ticket for the 2.3 release. Let's try to make it happen: > >>> https://issues.apache.org/jira/browse/IGNITE-6305 > >>> > >>> D. > >>> > >>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan < > >> dsetrak...@apache.org > >>> <javascript:;>> > >>> wrote: > >>> > >>>> > >>>> > >>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <raul....@evosent.com > >>> <javascript:;>> > >>>> wrote: > >>>> > >>>>> Yeah, I guess that's doable as well and requires less management > >> effort > >>>>> than my suggestion. We could use events [1] to store payload data > >> (e.g. > >>>>> IP, > >>>>> version, etc.) > >>>> > >>>> > >>>> Yes, we could use events or some other similar API provided by GA. > >>>> > >>>> > >>>>> What the download page CGI developed in? PHP? > >>>>> > >>>> > >>>> To be honest, no clue. I guess someone in the community can figure it > >>> out: > >>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html > >>>> > >>>> > >>>>> However, I'm not sure whether storing this data in a 3rd party > >> (Google) > >>> is > >>>>> compliant with the ASF policy. I guess it's no biggie, but if there's > >>>>> doubt > >>>>> in the PMC, it's better to ask legal@. > >>>> > >>>> > >>>> I am not sure there is anything to ask about. The whole Ignite website > >> is > >>>> GA enabled, and all we are doing is accessing a standard web page from > >>> the > >>>> Ignite web site. The information gathered from GA is available only to > >>> the > >>>> Ignite PMC. Frankly, I think legal@ will be very confused by this > >>>> question. > >>>> > >>>> Even ASF website itself uses GA: https://www.apache.org/ > >>>> foundation/policies/privacy.html > >>>> > >>>> > >>>>> I think Cos said it's OK; maybe Roman can pitch in. > >>>>> > >>>> > >>>> Sure, would be nice to hear from Roman as well. > >>>> > >>>> > >>>>> Cheers. > >>>>> > >>>>> [1] > >>>>> https://developers.google.com/analytics/devguides/collection > >>>>> /analyticsjs/events > >>>>> > >>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan < > >>>>> dsetrak...@apache.org <javascript:;>> > >>>>> wrote: > >>>>> > >>>>>> Raul, > >>>>>> > >>>>>> Could point about Javascript, it will not work because it executes > >> in > >>>>> the > >>>>>> browser. This means we need a server-side script, like CGI we are > >>> using > >>>>> on > >>>>>> our download page. > >>>>>> > >>>>>> How about this approach. We create something like ignite-version.cgi > >>>>> script > >>>>>> which will invoke a call to GA and then return the latest version. > >>>>>> > >>>>>> This should work, right? > >>>>>> > >>>>>> D. > >>>>>> > >>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani < > >> raul....@evosent.com > >>> <javascript:;>> > >>>>>> wrote: > >>>>>> > >>>>>>> Hey Dmitriy and all > >>>>>>> > >>>>>>> Also, since we have GA enabled for the website, we can track how > >>> many > >>>>>> times > >>>>>>>> this page was accessed, which will be equal to the number of > >>> starts. > >>>>>> This > >>>>>>>> way, the counter information is tracked and monitored by the > >>> Ignite > >>>>>> PMC. > >>>>>>> > >>>>>>> > >>>>>>> Unfortunately this won't work because GA is loaded via Javascript > >> on > >>>>> the > >>>>>>> browser, so Google will never receive the page hit. > >>>>>>> > >>>>>>> Given the constraints, the most viable solution is an HTTPS > >> endpoint > >>>>>>> running on ASF infra that Ignite invokes via a GET or POST > >> request. > >>>>> The > >>>>>>> simplest thing is to write each request in a log file, along with > >>> the > >>>>>>> timestamp, the version reported by the client, maybe the IP (not > >>> sure > >>>>>> about > >>>>>>> the ASF rules about this concerning privacy, I guess it's OK if > >> you > >>>>> make > >>>>>> it > >>>>>>> an opt-in) and a unique node identifier, i.e. a UUID the node > >>> creates > >>>>> on > >>>>>>> first startup or something. > >>>>>>> > >>>>>>> This endpoint would need some basic DDoS protection and > >> blacklisting > >>>>> to > >>>>>>> prevent data spoofing. > >>>>>>> > >>>>>>> If we'll be implementing this endpoint anyway, then there's no > >> point > >>>>>>> placing another file on Git or elsewhere for reporting the latest > >>>>>> versions: > >>>>>>> the endpoint itself can return them. > >>>>>>> > >>>>>>> WDYT? > >>>>>>> > >>>>>>> Cheers. > >>>>>>> > >>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan < > >>>>>> dsetrak...@apache.org <javascript:;>> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Cos, Raul, > >>>>>>>> > >>>>>>>> Thanks for the feedback. I completely agree about Maven Central > >>>>> being a > >>>>>>> 3rd > >>>>>>>> party repo (did not think about that initially). All your > >>>>> suggestions > >>>>>>> make > >>>>>>>> sense, but I would like to keep it as simple as possible, and so > >>> far > >>>>>>>> everything suggested required GIT dependencies and extra work. > >>>>>>>> > >>>>>>>> How about Yakov's suggestion. We simply add a page to the Ignite > >>>>>> website > >>>>>>>> which will have only the latest version. Every time a node > >> starts, > >>>>> it > >>>>>>>> receives the latest version from the page and suggests that > >> users > >>>>>> upgrade > >>>>>>>> if needed. > >>>>>>>> > >>>>>>>> Also, since we have GA enabled for the website, we can track how > >>>>> many > >>>>>>> times > >>>>>>>> this page was accessed, which will be equal to the number of > >>> starts. > >>>>>> This > >>>>>>>> way, the counter information is tracked and monitored by the > >>> Ignite > >>>>>> PMC. > >>>>>>>> > >>>>>>>> This approach looks pretty innocent to me and everything is kept > >>> and > >>>>>>>> managed within Apache. > >>>>>>>> > >>>>>>>> Thoughts? > >>>>>>>> > >>>>>>>> D. > >>>>>>>> > >>>>>>>> > >>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik < > >>>>> c...@apache.org <javascript:;>> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> I agree with Raul. > >>>>>>>>> - providing a ping-back address to a 3rd party might be frown > >>>>> upon by > >>>>>>>> some. > >>>>>>>>> And might have a consequences like stats collection about > >>> users' > >>>>>>>>> infrastructure. > >>>>>>>>> - checking an ASF git-repo is easy and won't download any > >> binary > >>>>>> data: > >>>>>>>>> everything is clear text and could be easily monitored by > >> any > >>> of > >>>>>> the > >>>>>>>>> network > >>>>>>>>> diagnostic tools, shall it be required. But it involves a > >> bit > >>> of > >>>>>> the > >>>>>>>>> release > >>>>>>>>> discipline. > >>>>>>>>> - the binary data download in the runtime is my main concern. > >>>>> This is > >>>>>>> the > >>>>>>>>> vector of MMA. What if someone gains the control over the > >>>>>> repository > >>>>>>>> and > >>>>>>>>> replaces the file with some malicious content. > >>>>>>>>> > >>>>>>>>> As for the particular mechanism: IIRC Ignite used to make a > >> call > >>>>> to > >>>>>> an > >>>>>>>>> external script to check upon the atest software version > >>> available > >>>>>> for > >>>>>>>>> download. In the past, the endpoint was running on a 3rd party > >>>>>> server, > >>>>>>> I > >>>>>>>>> believe the best way would be to put this script on ASF infra > >>> and > >>>>>> have > >>>>>>>> the > >>>>>>>>> "update checker" running in a completely controlled > >> environment. > >>>>>>>> Actually, > >>>>>>>>> I > >>>>>>>>> recall we had this very discussion during the Incubation; I > >> can > >>>>>>> probably > >>>>>>>>> dig > >>>>>>>>> out the corresponding thread. > >>>>>>>>> > >>>>>>>>> Thoughts? > >>>>>>>>> Cok > >>>>>>>>> > >>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote: > >>>>>>>>>> Hey guys > >>>>>>>>>> > >>>>>>>>>> In my opinion, maven.org is still owned by a third party > >>>>>> (Sonatype). > >>>>>>>> We > >>>>>>>>>> don't know what kind of data analysis or intelligence > >>> extraction > >>>>>> they > >>>>>>>>> run. > >>>>>>>>>> > >>>>>>>>>> If Ignite servers all over the world were hitting maven.org > >>>>>>>> periodically > >>>>>>>>>> asking for an Ignite Maven artifact, it gives Sonatype a > >> clear > >>>>>>>> indication > >>>>>>>>>> of who is running an Ignite server. > >>>>>>>>>> > >>>>>>>>>> They could reverse lookup the IP address and find out what > >>>>>>> corporation > >>>>>>>> it > >>>>>>>>>> is. > >>>>>>>>>> > >>>>>>>>>> How about having Ignite check the ASF Git directly? > >>>>>>>>>> > >>>>>>>>>> We could use the Git ssh API, but that would require a new > >>>>>>> dependency, > >>>>>>>>>> which I advise against. > >>>>>>>>>> > >>>>>>>>>> Alternatively, we could have it scrape this HTML for new Git > >>>>> tags: > >>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git > >>>>>>>>>> > >>>>>>>>>> Another option is to place a txt file in the root of the > >>> master > >>>>>>> branch > >>>>>>>>> (e.g > >>>>>>>>>> LATEST), containing a list of the latest GA versions for > >> each > >>>>> major > >>>>>>>>> version > >>>>>>>>>> line (1.x, 2.x). > >>>>>>>>>> > >>>>>>>>>> I would advocate this last option, but it requires somebody > >>>>>>> remembering > >>>>>>>>> to > >>>>>>>>>> update the file with every release, unless we automate it > >>> with a > >>>>>>> Maven > >>>>>>>>>> plugin. > >>>>>>>>>> > >>>>>>>>>> Hope that helps! > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dma...@apache.org > >>> <javascript:;>> wrote: > >>>>>>>>>> > >>>>>>>>>> I see nothing wrong with this approach. > >>>>>>>>>> > >>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what do you think? Is > >> it > >>>>> good > >>>>>>> to > >>>>>>>>> go? > >>>>>>>>>> > >>>>>>>>>> — > >>>>>>>>>> Denis > >>>>>>>>>> > >>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan < > >>>>>>>> dsetrak...@apache.org <javascript:;> > >>>>>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Is everyone OK with this approach? Should I file a ticket > >> on > >>>>> it? > >>>>>>>>>>> > >>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy Setrakyan < > >>>>>>>>> dsetrak...@apache.org <javascript:;>> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Igniters, > >>>>>>>>>>>> > >>>>>>>>>>>> There has been lots of talk of proposals about various > >>> usage > >>>>>>> metrics > >>>>>>>>> for > >>>>>>>>>>>> Ignite and nothing came of it. I would like to resurrect > >>> the > >>>>>> topic > >>>>>>>> and > >>>>>>>>>>>> propose something very simple and non-intrusive. > >>>>>>>>>>>> > >>>>>>>>>>>> 1. Update Checker > >>>>>>>>>>>> The main purpose of the update checker is not to collect > >>>>>> metrics, > >>>>>>>> but > >>>>>>>>> to > >>>>>>>>>>>> notify users about a new version of Ignite by accessing > >>>>>> maven.org > >>>>>>>> and > >>>>>>>>>>>> getting the version out of the metadata file: > >>>>>>>>>>>> http://repo2.maven.org/maven2/ > >>> org/apache/ignite/ignite-core/ > >>>>>>>>>>>> maven-metadata.xml > >>>>>>>>>>>> > >>>>>>>>>>>> This way we do not send any information anywhere and, at > >>> the > >>>>>> same > >>>>>>>>> time, > >>>>>>>>>>>> urge our users to download and start using the latest > >>>>> version of > >>>>>>>>> Ignite. > >>>>>>>>>>>> > >>>>>>>>>>>> 2. Startup Counter > >>>>>>>>>>>> This piece is optional, but we can also get an insight in > >>> how > >>>>>> many > >>>>>>>>> times > >>>>>>>>>> a > >>>>>>>>>>>> certain Ignite release gets started. This is just a cool > >>>>> metric > >>>>>>> for > >>>>>>>>> the > >>>>>>>>>>>> community to gauge the project popularity. You can think > >> of > >>>>> it > >>>>>> as > >>>>>>>> of a > >>>>>>>>>> page > >>>>>>>>>>>> visit counter shown on many websites. We can even decide > >> to > >>>>>>> display > >>>>>>>>> this > >>>>>>>>>>>> counter on the Ignite website as well. > >>>>>>>>>>>> > >>>>>>>>>>>> To do this, we can simply add a JAR in maven for every > >>>>> release, > >>>>>>> e.g. > >>>>>>>>>>>> ignite-start-counter.jar, which will contain only 1 byte. > >>>>> Every > >>>>>>> time > >>>>>>>>> an > >>>>>>>>>>>> Ignite node starts, it will download this JAR in the > >>>>> background. > >>>>>>>> Then > >>>>>>>>> we > >>>>>>>>>>>> will be able to view the number of the total downloads > >> for > >>>>> this > >>>>>>> JAR > >>>>>>>> in > >>>>>>>>>>>> Maven Central, which is essentially the number of starts > >> of > >>>>>> Ignite > >>>>>>>>> nodes. > >>>>>>>>>>>> > >>>>>>>>>>>> *Note that neither of the above suggestions require > >> Ignite > >>> to > >>>>>> send > >>>>>>>> or > >>>>>>>>>>>> track any user information whatsoever.* > >>>>>>>>>>>> > >>>>>>>>>>>> Please reply suggesting weather you are OK with this > >>>>> approach. > >>>>>>>>>>>> > >>>>>>>>>>>> D. > >>>>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>>> > >>> > >> > >
