[jira] [Commented] (TINKERPOP-1644) Improve script compilation syncronisation
[ https://issues.apache.org/jira/browse/TINKERPOP-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900158#comment-15900158 ] ASF GitHub Bot commented on TINKERPOP-1644: --- Github user spmallette commented on the issue: https://github.com/apache/tinkerpop/pull/567 I like this way better than the synchronization. I have some reservations about: ```java CompletableFuture.supplyAsync(() -> loader.parseClass(script, generateScriptName())); ``` That's going to use the `ForkJoinPool` to execute the compilation, which is a shared thread pool for the JVM that isn't configurable and basically outside of our control. Of course, getting rid of it means adding a new `ExecutorService` that we can configure and monitor in `GremlinGroovyScriptEngine`. Making that kind of change goes pretty deep into various aspects of TinkerPop. If @BrynCooke is ok with the changes I can merge what he has to a branch since it seems like we have general consensus on the approach and then make some additional changes and re-issue the PR for a final VOTE. If there's no objections I'll start work on that first thing tomorrow. > Improve script compilation syncronisation > - > > Key: TINKERPOP-1644 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1644 > Project: TinkerPop > Issue Type: Improvement > Components: groovy >Affects Versions: 3.2.4 >Reporter: Bryn Cooke > > Currently there is no synchronisation around script compilation. This means > that if a particularly heavy script is in use, many threads may end up > compiling the same script. > It would seem like a good idea to have some some sort of synchronisation to > prevent ever getting to this stage. > In addition, there will be cases where users will repeatedly submit broken > scripts to the server. In this case it is useful to log the error the first > time the script compilation is attempted and then cache the error for > subsequent runs. > Finally I have found some scripts take in excess of 30 seconds to compile. To > aid performance debugging the script compilation times should be included in > the logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] tinkerpop issue #567: TINKERPOP-1644 Improve script compilation syncronisati...
Github user spmallette commented on the issue: https://github.com/apache/tinkerpop/pull/567 I like this way better than the synchronization. I have some reservations about: ```java CompletableFuture.supplyAsync(() -> loader.parseClass(script, generateScriptName())); ``` That's going to use the `ForkJoinPool` to execute the compilation, which is a shared thread pool for the JVM that isn't configurable and basically outside of our control. Of course, getting rid of it means adding a new `ExecutorService` that we can configure and monitor in `GremlinGroovyScriptEngine`. Making that kind of change goes pretty deep into various aspects of TinkerPop. If @BrynCooke is ok with the changes I can merge what he has to a branch since it seems like we have general consensus on the approach and then make some additional changes and re-issue the PR for a final VOTE. If there's no objections I'll start work on that first thing tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1644) Improve script compilation syncronisation
[ https://issues.apache.org/jira/browse/TINKERPOP-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899809#comment-15899809 ] ASF GitHub Bot commented on TINKERPOP-1644: --- Github user okram commented on the issue: https://github.com/apache/tinkerpop/pull/567 Yes. `computeIfAbsent()` is much better than synchronized `get()`, check, `put()`. > Improve script compilation syncronisation > - > > Key: TINKERPOP-1644 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1644 > Project: TinkerPop > Issue Type: Improvement > Components: groovy >Affects Versions: 3.2.4 >Reporter: Bryn Cooke > > Currently there is no synchronisation around script compilation. This means > that if a particularly heavy script is in use, many threads may end up > compiling the same script. > It would seem like a good idea to have some some sort of synchronisation to > prevent ever getting to this stage. > In addition, there will be cases where users will repeatedly submit broken > scripts to the server. In this case it is useful to log the error the first > time the script compilation is attempted and then cache the error for > subsequent runs. > Finally I have found some scripts take in excess of 30 seconds to compile. To > aid performance debugging the script compilation times should be included in > the logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] tinkerpop issue #567: TINKERPOP-1644 Improve script compilation syncronisati...
Github user okram commented on the issue: https://github.com/apache/tinkerpop/pull/567 Yes. `computeIfAbsent()` is much better than synchronized `get()`, check, `put()`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1644) Improve script compilation syncronisation
[ https://issues.apache.org/jira/browse/TINKERPOP-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899808#comment-15899808 ] ASF GitHub Bot commented on TINKERPOP-1644: --- Github user dkuppitz commented on the issue: https://github.com/apache/tinkerpop/pull/567 This part: ``` if (clazz != null) { return clazz.get(); } clazz = CompletableFuture.supplyAsync(() -> loader.parseClass(script, generateScriptName())); classMap.put(script, clazz); return clazz.get(); ``` should be more like: ``` classMap.computeIfAbsent(script, s -> CompletableFuture.supplyAsync(() -> loader.parseClass(s, generateScriptName( return classMap.get(script).get(); ``` No? > Improve script compilation syncronisation > - > > Key: TINKERPOP-1644 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1644 > Project: TinkerPop > Issue Type: Improvement > Components: groovy >Affects Versions: 3.2.4 >Reporter: Bryn Cooke > > Currently there is no synchronisation around script compilation. This means > that if a particularly heavy script is in use, many threads may end up > compiling the same script. > It would seem like a good idea to have some some sort of synchronisation to > prevent ever getting to this stage. > In addition, there will be cases where users will repeatedly submit broken > scripts to the server. In this case it is useful to log the error the first > time the script compilation is attempted and then cache the error for > subsequent runs. > Finally I have found some scripts take in excess of 30 seconds to compile. To > aid performance debugging the script compilation times should be included in > the logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] tinkerpop issue #567: TINKERPOP-1644 Improve script compilation syncronisati...
Github user dkuppitz commented on the issue: https://github.com/apache/tinkerpop/pull/567 This part: ``` if (clazz != null) { return clazz.get(); } clazz = CompletableFuture.supplyAsync(() -> loader.parseClass(script, generateScriptName())); classMap.put(script, clazz); return clazz.get(); ``` should be more like: ``` classMap.computeIfAbsent(script, s -> CompletableFuture.supplyAsync(() -> loader.parseClass(s, generateScriptName( return classMap.get(script).get(); ``` No? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1644) Improve script compilation syncronisation
[ https://issues.apache.org/jira/browse/TINKERPOP-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899793#comment-15899793 ] ASF GitHub Bot commented on TINKERPOP-1644: --- Github user robertdale commented on the issue: https://github.com/apache/tinkerpop/pull/567 VOTE +1 > Improve script compilation syncronisation > - > > Key: TINKERPOP-1644 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1644 > Project: TinkerPop > Issue Type: Improvement > Components: groovy >Affects Versions: 3.2.4 >Reporter: Bryn Cooke > > Currently there is no synchronisation around script compilation. This means > that if a particularly heavy script is in use, many threads may end up > compiling the same script. > It would seem like a good idea to have some some sort of synchronisation to > prevent ever getting to this stage. > In addition, there will be cases where users will repeatedly submit broken > scripts to the server. In this case it is useful to log the error the first > time the script compilation is attempted and then cache the error for > subsequent runs. > Finally I have found some scripts take in excess of 30 seconds to compile. To > aid performance debugging the script compilation times should be included in > the logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] tinkerpop issue #567: TINKERPOP-1644 Improve script compilation syncronisati...
Github user robertdale commented on the issue: https://github.com/apache/tinkerpop/pull/567 VOTE +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1644) Improve script compilation syncronisation
[ https://issues.apache.org/jira/browse/TINKERPOP-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899603#comment-15899603 ] ASF GitHub Bot commented on TINKERPOP-1644: --- Github user BrynCooke commented on the issue: https://github.com/apache/tinkerpop/pull/567 I like the idea of using a Future. > Improve script compilation syncronisation > - > > Key: TINKERPOP-1644 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1644 > Project: TinkerPop > Issue Type: Improvement > Components: groovy >Affects Versions: 3.2.4 >Reporter: Bryn Cooke > > Currently there is no synchronisation around script compilation. This means > that if a particularly heavy script is in use, many threads may end up > compiling the same script. > It would seem like a good idea to have some some sort of synchronisation to > prevent ever getting to this stage. > In addition, there will be cases where users will repeatedly submit broken > scripts to the server. In this case it is useful to log the error the first > time the script compilation is attempted and then cache the error for > subsequent runs. > Finally I have found some scripts take in excess of 30 seconds to compile. To > aid performance debugging the script compilation times should be included in > the logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] tinkerpop issue #567: TINKERPOP-1644 Improve script compilation syncronisati...
Github user BrynCooke commented on the issue: https://github.com/apache/tinkerpop/pull/567 I like the idea of using a Future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1644) Improve script compilation syncronisation
[ https://issues.apache.org/jira/browse/TINKERPOP-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899552#comment-15899552 ] ASF GitHub Bot commented on TINKERPOP-1644: --- Github user robertdale commented on the issue: https://github.com/apache/tinkerpop/pull/567 That sounds like a good alternative. > Improve script compilation syncronisation > - > > Key: TINKERPOP-1644 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1644 > Project: TinkerPop > Issue Type: Improvement > Components: groovy >Affects Versions: 3.2.4 >Reporter: Bryn Cooke > > Currently there is no synchronisation around script compilation. This means > that if a particularly heavy script is in use, many threads may end up > compiling the same script. > It would seem like a good idea to have some some sort of synchronisation to > prevent ever getting to this stage. > In addition, there will be cases where users will repeatedly submit broken > scripts to the server. In this case it is useful to log the error the first > time the script compilation is attempted and then cache the error for > subsequent runs. > Finally I have found some scripts take in excess of 30 seconds to compile. To > aid performance debugging the script compilation times should be included in > the logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] tinkerpop issue #567: TINKERPOP-1644 Improve script compilation syncronisati...
Github user robertdale commented on the issue: https://github.com/apache/tinkerpop/pull/567 That sounds like a good alternative. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1644) Improve script compilation syncronisation
[ https://issues.apache.org/jira/browse/TINKERPOP-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899481#comment-15899481 ] ASF GitHub Bot commented on TINKERPOP-1644: --- Github user dkuppitz commented on the issue: https://github.com/apache/tinkerpop/pull/567 Hmm, I haven't done any benchmarks, but this seem to be valid points. Just a thought: Would it help to have the class map be like `ManagedConcurrentValueMapclassMap`? > Improve script compilation syncronisation > - > > Key: TINKERPOP-1644 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1644 > Project: TinkerPop > Issue Type: Improvement > Components: groovy >Affects Versions: 3.2.4 >Reporter: Bryn Cooke > > Currently there is no synchronisation around script compilation. This means > that if a particularly heavy script is in use, many threads may end up > compiling the same script. > It would seem like a good idea to have some some sort of synchronisation to > prevent ever getting to this stage. > In addition, there will be cases where users will repeatedly submit broken > scripts to the server. In this case it is useful to log the error the first > time the script compilation is attempted and then cache the error for > subsequent runs. > Finally I have found some scripts take in excess of 30 seconds to compile. To > aid performance debugging the script compilation times should be included in > the logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] tinkerpop issue #567: TINKERPOP-1644 Improve script compilation syncronisati...
Github user dkuppitz commented on the issue: https://github.com/apache/tinkerpop/pull/567 Hmm, I haven't done any benchmarks, but this seem to be valid points. Just a thought: Would it help to have the class map be like `ManagedConcurrentValueMapclassMap`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1645) Bump to Groovy 2.4.9
[ https://issues.apache.org/jira/browse/TINKERPOP-1645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899454#comment-15899454 ] ASF GitHub Bot commented on TINKERPOP-1645: --- Github user robertdale commented on the issue: https://github.com/apache/tinkerpop/pull/568 VOTE +1 > Bump to Groovy 2.4.9 > > > Key: TINKERPOP-1645 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1645 > Project: TinkerPop > Issue Type: Improvement > Components: groovy >Affects Versions: 3.1.6 >Reporter: stephen mallette >Assignee: stephen mallette > > Groovy released 2.4.9 in the last week or so. It's a small release with > basically all bug fixes. One fix that might be important to TinkerPop is this > one: GROOVY-8067 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] tinkerpop issue #568: TINKERPOP-1645 Bump to groovy 2.4.9
Github user robertdale commented on the issue: https://github.com/apache/tinkerpop/pull/568 VOTE +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1644) Improve script compilation syncronisation
[ https://issues.apache.org/jira/browse/TINKERPOP-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899434#comment-15899434 ] ASF GitHub Bot commented on TINKERPOP-1644: --- Github user robertdale commented on the issue: https://github.com/apache/tinkerpop/pull/567 I have a concern on overall performance. I agree that "Script compilation is synchronised" helps in the rare case when two or more identical scripts are submitted concurrently. However, in the more likely case of new, unique scripts being submitted concurrently, it would seem that this effectively serializes script compilation creating a bottleneck. Unless I'm missing something, parallel compilation seems to be the better choice. > Improve script compilation syncronisation > - > > Key: TINKERPOP-1644 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1644 > Project: TinkerPop > Issue Type: Improvement > Components: groovy >Affects Versions: 3.2.4 >Reporter: Bryn Cooke > > Currently there is no synchronisation around script compilation. This means > that if a particularly heavy script is in use, many threads may end up > compiling the same script. > It would seem like a good idea to have some some sort of synchronisation to > prevent ever getting to this stage. > In addition, there will be cases where users will repeatedly submit broken > scripts to the server. In this case it is useful to log the error the first > time the script compilation is attempted and then cache the error for > subsequent runs. > Finally I have found some scripts take in excess of 30 seconds to compile. To > aid performance debugging the script compilation times should be included in > the logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] tinkerpop issue #567: TINKERPOP-1644 Improve script compilation syncronisati...
Github user robertdale commented on the issue: https://github.com/apache/tinkerpop/pull/567 I have a concern on overall performance. I agree that "Script compilation is synchronised" helps in the rare case when two or more identical scripts are submitted concurrently. However, in the more likely case of new, unique scripts being submitted concurrently, it would seem that this effectively serializes script compilation creating a bottleneck. Unless I'm missing something, parallel compilation seems to be the better choice. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
RE: [GitHub] tinkerpop issue #567: TINKERPOP-1644 Improve script compilation syncronisati...
This code is using the double-checked locking idiom, which does not work in Java: Class clazz = getScriptClassFromMap(script); if (clazz != null) { return clazz; } synchronized (this) { clazz = getScriptClassFromMap(script); if (clazz != null) { return clazz; } See https://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html I don't know the details well enough to judge whether the failure modes will be harmful in this case. I think it's best avoided or replaced with one of the suggestions in the link above. -Paul -Original Message- From: spmallette [mailto:g...@git.apache.org] Sent: Tuesday, March 07, 2017 6:25 AM To: dev@tinkerpop.apache.org Subject: [GitHub] tinkerpop issue #567: TINKERPOP-1644 Improve script compilation syncronisati... Github user spmallette commented on the issue: https://github.com/apache/tinkerpop/pull/567 I will clean up a few code formatting things on merge, but this looks good to me. All tests pass with `docker/build.sh -t -n -i` VOTE +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1644) Improve script compilation syncronisation
[ https://issues.apache.org/jira/browse/TINKERPOP-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899374#comment-15899374 ] ASF GitHub Bot commented on TINKERPOP-1644: --- Github user dkuppitz commented on the issue: https://github.com/apache/tinkerpop/pull/567 Yep, missing some spaces here and there, but the code in general looks good. VOTE: +1 > Improve script compilation syncronisation > - > > Key: TINKERPOP-1644 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1644 > Project: TinkerPop > Issue Type: Improvement > Components: groovy >Affects Versions: 3.2.4 >Reporter: Bryn Cooke > > Currently there is no synchronisation around script compilation. This means > that if a particularly heavy script is in use, many threads may end up > compiling the same script. > It would seem like a good idea to have some some sort of synchronisation to > prevent ever getting to this stage. > In addition, there will be cases where users will repeatedly submit broken > scripts to the server. In this case it is useful to log the error the first > time the script compilation is attempted and then cache the error for > subsequent runs. > Finally I have found some scripts take in excess of 30 seconds to compile. To > aid performance debugging the script compilation times should be included in > the logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] tinkerpop issue #567: TINKERPOP-1644 Improve script compilation syncronisati...
Github user dkuppitz commented on the issue: https://github.com/apache/tinkerpop/pull/567 Yep, missing some spaces here and there, but the code in general looks good. VOTE: +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TINKERPOP-1644) Improve script compilation syncronisation
[ https://issues.apache.org/jira/browse/TINKERPOP-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899283#comment-15899283 ] ASF GitHub Bot commented on TINKERPOP-1644: --- Github user spmallette commented on the issue: https://github.com/apache/tinkerpop/pull/567 I will clean up a few code formatting things on merge, but this looks good to me. All tests pass with `docker/build.sh -t -n -i` VOTE +1 > Improve script compilation syncronisation > - > > Key: TINKERPOP-1644 > URL: https://issues.apache.org/jira/browse/TINKERPOP-1644 > Project: TinkerPop > Issue Type: Improvement > Components: groovy >Affects Versions: 3.2.4 >Reporter: Bryn Cooke > > Currently there is no synchronisation around script compilation. This means > that if a particularly heavy script is in use, many threads may end up > compiling the same script. > It would seem like a good idea to have some some sort of synchronisation to > prevent ever getting to this stage. > In addition, there will be cases where users will repeatedly submit broken > scripts to the server. In this case it is useful to log the error the first > time the script compilation is attempted and then cache the error for > subsequent runs. > Finally I have found some scripts take in excess of 30 seconds to compile. To > aid performance debugging the script compilation times should be included in > the logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] tinkerpop issue #567: TINKERPOP-1644 Improve script compilation syncronisati...
Github user spmallette commented on the issue: https://github.com/apache/tinkerpop/pull/567 I will clean up a few code formatting things on merge, but this looks good to me. All tests pass with `docker/build.sh -t -n -i` VOTE +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---