[GitHub] zeppelin issue #2296: [ZEPPELIN-2463] Avoid Locking interpreterSettings duri...
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2296 @Leemoonsoo , Could you please take a look at this change ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2310: ZEPPELIN-2484 do a Null check before calling tr...
GitHub user benoyantony opened a pull request: https://github.com/apache/zeppelin/pull/2310 ZEPPELIN-2484 do a Null check before calling trim on paragraph's text ### What is this PR for? Prevent NullPointerException during check to determine whether a new paragraph needs to added. The fix is to switch order of null check and trim operation so that null check is performed before trim() ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2484 ### How should this be tested? This can be tested with a hive interpreter Create a note and add a paragraph with some query in it. Run all paragraphs. A new paragraph is automatically added. Run all paragraphs again. A NullPointerException is logged in the logs. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No You can merge this pull request into a Git repository by running: $ git pull https://github.com/benoyantony/zeppelin ZEPPELIN-2484 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2310.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2310 commit 09e956e0ccc7d5522857fdece5d2232239085f65 Author: Benoy Antony <be...@apache.org> Date: 2017-05-02T22:17:44Z ZEPPELIN-2484 do a Null check before calling trim on paragraph's text --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2301: [ZEPPELIN-1625] Override Interpreter settings at User ...
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2301 @zjffdu , Thanks for pointing me to ZEPPELIN-1338. I did not know about it. I believe, the intent is same. I agree that the approach is different. We wanted to shield the user from interpreter settings. The approach in this change is to specify a property name and default value in the interpreter setting. Users can set a value for the property. When the interpreter is created, then the user's property value is used as the interpreter setting. User's properties are persisted. One advantage of this approach is to use the same property across multiple interpreter settings. We have been using this feature at our company for last 3 months and its been effective. I'll review ZEPPELIN-1338 carefully to understand the approach mentioned in that. Thanks again for letting me know of the related effort. Its good to know that its a feature needed by many. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2296: [ZEPPELIN-2463] Avoid Locking interpreterSettings duri...
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2296 @prabhjyotsingh , Could you please review this change? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2301: [ZEPPELIN-1625] Override Interpreter settings at User ...
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2301 @Leemoonsoo , Could you please review this feature ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2297: [ZEPPELIN-2465] Minor code fixes for the livy package
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2297 @zjffdu , could you please review this change ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2301: [ZEPPELIN-1625] Override Interpreter settings a...
GitHub user benoyantony opened a pull request: https://github.com/apache/zeppelin/pull/2301 [ZEPPELIN-1625] Override Interpreter settings at User level ### What is this PR for? When multiple users share a zeppelin instance, it is sometimes desirable to use the same interpreter with different settings. It should be possible an administrator to setup some of the Interpreter settings as overridable by setting a place holder and a default value It should be possible for a user to specify the actual values via properties. ### What type of PR is it? Feature ### Todos Override interpreter setting at notebook level ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1625 ### How should this be tested? Unit tests are added. Please see screen shots for manual testing ### Screenshots   ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? Yes , added You can merge this pull request into a Git repository by running: $ git pull https://github.com/benoyantony/zeppelin user_properties Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2301.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2301 commit 6972b7cf094ef07a2a1e295169966656f638f361 Author: Benoy Antony <be...@apache.org> Date: 2017-04-28T22:08:45Z ZEPPELIN-1625 Override Interpreter settings at User level --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1623: ZEPPELIN-1625 add support for user specific pro...
Github user benoyantony closed the pull request at: https://github.com/apache/zeppelin/pull/1623 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2297: [ZEPPELIN-2465] Minor code fixes for the livy p...
GitHub user benoyantony opened a pull request: https://github.com/apache/zeppelin/pull/2297 [ZEPPELIN-2465] Minor code fixes for the livy package ### What is this PR for? Minor code fixes for the livy package. The code fixes include : Fixing a typo in a classname - BaseLivyInterprereter to BaseLivyInterpreter Removing an unused variable in BaseLivyInterpreter Removing unused imports in a few classes ### What type of PR is it? Refactoring ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2465 ### How should this be tested? No need to test as there is no change in funcionality ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? NO You can merge this pull request into a Git repository by running: $ git pull https://github.com/benoyantony/zeppelin refactor Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2297.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2297 commit 74016ad7aea693df56e1f661afcd875bf548191b Author: Benoy Antony <be...@apache.org> Date: 2017-04-28T06:51:48Z ZEPPELIN-2465 Minor code fixes for the livy package --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2296: [ZEPPELIN-2463] Avoid Locking interpreterSettin...
GitHub user benoyantony opened a pull request: https://github.com/apache/zeppelin/pull/2296 [ZEPPELIN-2463] Avoid Locking interpreterSettings during Notebook deletion ### What is this PR for? Deletion of a notebook requires locking interpreterSettings. If the deletion is delayed , then lock is not released. At that point, we cannot run any notebook because everything is waiting to lock interpreterSettings. Looking at the code, there is no reason to lock the InterpreterSettings object in InterpreterSettingManager.removeNoteInterpreterSettingBinding. Similarly in InterpreterSettingManager.getNoteInterpreterSettingBinding only interpreterSettingBinding is accessed and its already a thread safe object. So we can remove synchronization on InterpreterSettings ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2463 ### How should this be tested? Being a concurrency issue, it is difficult to test. Please see the comments in the jira to see the issue experienced on a production zeppelin server. ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? NO You can merge this pull request into a Git repository by running: $ git pull https://github.com/benoyantony/zeppelin ZEPPELIN-2463 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2296.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2296 commit ffe40b4e99e75ea1c298bad7f9383874f5c54678 Author: Benoy Antony <be...@apache.org> Date: 2017-04-28T06:27:33Z ZEPPELIN-2463 Avoid Locking interpreterSettings during Notebook deletion --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2295: [ZEPPELIN-2463] Avoid Locking interpreterSettin...
Github user benoyantony closed the pull request at: https://github.com/apache/zeppelin/pull/2295 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2295: [ZEPPELIN-2463] Avoid Locking interpreterSettin...
GitHub user benoyantony opened a pull request: https://github.com/apache/zeppelin/pull/2295 [ZEPPELIN-2463] Avoid Locking interpreterSettings during Notebook deletion ### What is this PR for? Deletion of a notebook requires locking interpreterSettings. If the deletion is delayed , then lock is not released. At that point, we cannot run any notebook because everything is waiting to lock interpreterSettings. Looking at the code, there is no reason to lock the InterpreterSettings object in InterpreterSettingManager.removeNoteInterpreterSettingBinding. Similarly in InterpreterSettingManager.getNoteInterpreterSettingBinding only interpreterSettingBinding is accessed and its already a thread safe object. So we can remove synchronization on InterpreterSettings ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2463 ### How should this be tested? Being a concurrency issue, it is difficult to test this. ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? NO You can merge this pull request into a Git repository by running: $ git pull https://github.com/benoyantony/zeppelin ZEPPELIN-2463 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2295.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2295 commit 5fbe6d4cbd9cbf0581e5d303cd5e4005268c2c2a Author: Benoy Antony <be...@apache.org> Date: 2017-04-28T05:58:09Z ZEPPELIN-2463 Avoid Locking interpreterSettings during Notebook deletion --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2223: [ZEPPELIN-2355] Livy cancel enhancements
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2223 Thanks @zjffdu for reviewing and committing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2223: [ZEPPELIN-2355] Livy cancel enhancements
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2223 A testcase errored out and it seems to be unrelated. ParagraphActionsIT.testRunOnSelectionChange --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2223: [ZEPPELIN-2355] Livy cancel enhancements
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2223 @zjffdu , Will try to add the unit test for cancellation in LivySparkSQLInterpreter. Any suggestion on how to simulate a sleep in SQL ? Also, where are the other comments ? Sorry, I am not very familiar with the github code reviews. It may be very obvious. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2223: [ZEPPELIN-2355] Livy cancel enhancements
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2223 @zjffdu , Could you please review ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2223: [ZEPPELIN-2355] Livy cancel enhancements
GitHub user benoyantony opened a pull request: https://github.com/apache/zeppelin/pull/2223 [ZEPPELIN-2355] Livy cancel enhancements ### What is this PR for? The Cancel functionality for the Livy interpreter has a couple of issues. One issue is because a variable is not published correctly. Second issue is observed when there is a delay in launching the application. Any cancel before application launch is ignored. The third issue is that Cancel is not correctly implemented for SparkSQLInterpreter. ### What type of PR is it? Bug Fix ### Todos * [ ] - Task ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2355 ### How should this be tested? The test cases are modified to test the changes. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No You can merge this pull request into a Git repository by running: $ git pull https://github.com/benoyantony/zeppelin livy-cancel-enhancement Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2223.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2223 commit 8673acfd5da76e547b986fb9dcfdb74fc3c379cf Author: Benoy Antony <be...@apache.org> Date: 2017-04-05T03:27:30Z ZEPPELIN-2355 Fix race conditions while cancelling a paragraph commit 4f1f243b2b8e16860b25dd59f31924fb2c16c8c3 Author: Benoy Antony <be...@apache.org> Date: 2017-04-05T03:36:16Z remove unrelated changes in imports --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2201: [ZEPPELIN-1965] Livy SQL Interpreter: Should use df.sh...
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2201 Thank you @zjffdu and @felixcheung --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2201: [ZEPPELIN-1965] Livy SQL Interpreter: Should use df.sh...
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2201 It should not break anything as the property name remains same as before. The documentation had a typo in it. It was missing 'sql' in it. If it's confusing, I can do it as a separate fix. On Mar 31, 2017 8:48 PM, "Felix Cheung" <notificati...@github.com> wrote: > *@felixcheung* commented on this pull request. > -- > > In docs/interpreter/livy.md > <https://github.com/apache/zeppelin/pull/2201#discussion_r109276567>: > > > @@ -56,14 +56,14 @@ Example: `spark.driver.memory` to `livy.spark.driver.memory` > URL where livy server is running > > > -zeppelin.livy.spark.maxResult > +zeppelin.livy.spark.sql.maxResult > > I love this, but it might break existing users. Perhaps we don't change > this for now? > > â > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/zeppelin/pull/2201#pullrequestreview-30373736>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AB1uyu_U3xz4WnpFAGISqFZMWZSy3q8Jks5rrckIgaJpZM4Msjp1> > . > --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2201: [ZEPPELIN-1965] Livy SQL Interpreter: Should use df.sh...
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2201 Thanks for the prompt review, @zjffdu . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2201: [ZEPPELIN-1965] Livy SQL Interpreter: Should use df.sh...
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2201 That seems to have worked. thanks @zjffdu . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #2201: [ZEPPELIN-1965] Livy SQL Interpreter: Should use df.sh...
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/2201 There seems to be a problem with Jenkins as the error is " java.io.IOException: Remote call on ubuntu-2 failed". Any idea on howto overcome this ? The travis build has all checks passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #2201: [ZEPPELIN-1965] Livy SQL Interpreter: Should us...
GitHub user benoyantony opened a pull request: https://github.com/apache/zeppelin/pull/2201 [ZEPPELIN-1965] Livy SQL Interpreter: Should use df.show(1000, false)⦠⦠to display results ### What is this PR for? Livy SQL interpreter truncate result strings of size greater than 20. In some cases, we like to see the full string. We are adding an interpreter property zeppelin.livy.spark.sql.truncate to control whether to truncate strings or not. By default, zeppelin.livy.spark.sql.truncate is set to true. ### What type of PR is it? Improvement ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1965 ### How should this be tested? Set zeppelin.livy.spark.sql.truncate to true or false Run a SQL query which produces string values of length greater than 20. Depending on the value of zeppelin.livy.spark.sql.truncate, the strings will either get truncated or not. ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No You can merge this pull request into a Git repository by running: $ git pull https://github.com/benoyantony/zeppelin master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2201.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2201 commit 3a3a4e9ce01f4dc184175e1f22b77d7a3bb6b46b Author: Benoy Antony <be...@apache.org> Date: 2017-03-29T06:24:31Z [ZEPPELIN-1965] Livy SQL Interpreter: Should use df.show(1000, false) to display results --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin issue #1778: ZEPPELIN-1834 Deadlock in Zeppelin when running multip...
Github user benoyantony commented on the issue: https://github.com/apache/zeppelin/pull/1778 When a note is run as by scheduler, thread T1 calls Note.runAll() and lock on the paragraphs is acquired. Later, T1 tries to obtain locks on paragraphs of other notes as part of status update. If scheduler starts running another note using a different thread, this sequence will result in deadlock. T1 -> locks N1.paragraphs , needs to obtain N2.paragraphs as part of broadcastUpdateNotebookJobInfo T2-> locks N2.paragraphs , needs to obtain N1.paragraphs as part of broadcastUpdateNotebookJobInfo T1 is waiting for N2.paragraphs holding lock on N1.paragraphs. T2 is waiting for N1.paragraphs holding lock on N2.paragraphs. The relevant thread dump is below: "DefaultQuartzScheduler_Worker-6": at org.apache.zeppelin.notebook.Note.getParagraphs(Note.java:537) - waiting to lock <0xc06cb168> (a java.util.ArrayList) at org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime(Notebook.java:734) at org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNotebookJobInfo(NotebookServer.java:423) at org.apache.zeppelin.socket.NotebookServer$ParagraphListenerImpl.afterStatusChange(NotebookServer.java:1468) at org.apache.zeppelin.scheduler.Job.setStatus(Job.java:150) at org.apache.zeppelin.scheduler.RemoteScheduler.submit(RemoteScheduler.java:148) at org.apache.zeppelin.notebook.Note.run(Note.java:510) at org.apache.zeppelin.notebook.Note.runAll(Note.java:484) - locked <0xc071ad00> (a java.util.ArrayList) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:786) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0xc0864798> (a java.lang.Object) "DefaultQuartzScheduler_Worker-7": at org.apache.zeppelin.notebook.Note.getParagraphs(Note.java:537) - waiting to lock <0xc071ad00> (a java.util.ArrayList) at org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime(Notebook.java:734) at org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNotebookJobInfo(NotebookServer.java:423) at org.apache.zeppelin.socket.NotebookServer$ParagraphListenerImpl.afterStatusChange(NotebookServer.java:1468) at org.apache.zeppelin.scheduler.Job.setStatus(Job.java:150) at org.apache.zeppelin.scheduler.RemoteScheduler.submit(RemoteScheduler.java:148) at org.apache.zeppelin.notebook.Note.run(Note.java:510) at org.apache.zeppelin.notebook.Note.runAll(Note.java:484) - locked <0xc06cb168> (a java.util.ArrayList) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:786) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0xc0864a98> (a java.lang.Object) Found 1 deadlock. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zeppelin pull request #1778: ZEPPELIN-1834 Deadlock in Zeppelin when running...
GitHub user benoyantony opened a pull request: https://github.com/apache/zeppelin/pull/1778 ZEPPELIN-1834 Deadlock in Zeppelin when running multiple notes via sc⦠### What is this PR for? What is this PR for? To prevent deadlock when different notes are run simultaneously by the scheduler. When a note is run as by scheduler, thread T1 calls Note.runAll() and lock on the paragraphs is acquired. Later, T1 tries to obtain locks on paragraphs of other notes as part of status update. If scheduler starts running another note using a different thread, this sequence will result in deadlock. T1 -> locks N1.paragraphs , needs to obtain N2.paragraphs as part of broadcastUpdateNotebookJobInfo T2-> locks N2.paragraphs , needs to obtain N1.paragraphs as part of broadcastUpdateNotebookJobInfo T1 is waiting for N2.paragraphs holding lock on N1.paragraphs. T2 is waiting for N1.paragraphs holding lock on N2.paragraphs. The relevant thread dump is below: "DefaultQuartzScheduler_Worker-6": at org.apache.zeppelin.notebook.Note.getParagraphs(Note.java:537) - waiting to lock <0xc06cb168> (a java.util.ArrayList) at org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime(Notebook.java:734) at org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNotebookJobInfo(NotebookServer.java:423) at org.apache.zeppelin.socket.NotebookServer$ParagraphListenerImpl.afterStatusChange(NotebookServer.java:1468) at org.apache.zeppelin.scheduler.Job.setStatus(Job.java:150) at org.apache.zeppelin.scheduler.RemoteScheduler.submit(RemoteScheduler.java:148) at org.apache.zeppelin.notebook.Note.run(Note.java:510) at org.apache.zeppelin.notebook.Note.runAll(Note.java:484) - locked <0xc071ad00> (a java.util.ArrayList) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:786) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0xc0864798> (a java.lang.Object) "DefaultQuartzScheduler_Worker-7": at org.apache.zeppelin.notebook.Note.getParagraphs(Note.java:537) - waiting to lock <0xc071ad00> (a java.util.ArrayList) at org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime(Notebook.java:734) at org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNotebookJobInfo(NotebookServer.java:423) at org.apache.zeppelin.socket.NotebookServer$ParagraphListenerImpl.afterStatusChange(NotebookServer.java:1468) at org.apache.zeppelin.scheduler.Job.setStatus(Job.java:150) at org.apache.zeppelin.scheduler.RemoteScheduler.submit(RemoteScheduler.java:148) at org.apache.zeppelin.notebook.Note.run(Note.java:510) at org.apache.zeppelin.notebook.Note.runAll(Note.java:484) - locked <0xc06cb168> (a java.util.ArrayList) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:786) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0xc0864a98> (a java.lang.Object) Found 1 deadlock. ### What type of PR is it? Bug Fix ### Todos ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1834 [ZEPPELIN-1834] ### How should this be tested? Manual testing steps: * â¨Create 2 notes. * Set a scheduler on each of them to run at the same time. * Deadlock will occur and Zeppelin hangs. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? NO â¦heduler You can merge this pull request into a Git repository by running: $ git pull https://github.com/benoyantony/zeppelin deadlock Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/1778.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1778 commit 96e3cfb3bab8544d474bfbb0077d6566ff9e22ca Author: Benoy Antony <be...@apache.org> Date: 2016-12-16T21:28:38Z ZEPPELIN-1834 Deadlock in Zeppelin when running multiple notes via scheduler --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, pleas
[GitHub] zeppelin issue #1623: ZEPPELIN-1625 add support for user specific properties
Github user benoyantony commented on the issue:
https://github.com/apache/zeppelin/pull/1623
thanks @1ambda , Sorry for the delay.
1. If this feature is merged, then the properties can be referred in the
interpreter properties. Each user can override the property value.
2. Once the feature is merged, I will have another pull request on the Livy
Interpreter which make use of the feature for its properties.
I have tested this feature at our company. Attached is screenshot where
the administrator setup the Livy property - livy.spark.yarn.queue to
${queue,defaultQueue}
In the second screenshot, the user override the queue as benoyqueue. If
the user does not override, then the property will remain as defaultQueue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zeppelin pull request #1611: ZEPPELIN-1625 - Add support of user specific pr...
GitHub user benoyantony opened a pull request: https://github.com/apache/zeppelin/pull/1611 ZEPPELIN-1625 - Add support of user specific properties ### What is this PR for? User should be able to define properties and these properties must be passed to Interpreter via Interpreter Context. Add functionality so that Interpreter can substitute the its property value with the value specified by the user. ### What type of PR is it? Feature ### Todos ### What is the Jira issue? ZEPPELIN-1625 ### How should this be tested? unit tests provided ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes You can merge this pull request into a Git repository by running: $ git pull https://github.com/benoyantony/zeppelin master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/1611.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1611 commit 971a431de5a4373ae43a0bf664605475298eff78 Author: Benoy Antony <be...@apache.org> Date: 2016-11-07T22:53:42Z ZEPPELIN-1625 - Add support of user specific properties --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
