[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Attachment: PIG-1313-4.patch
# Attaching PIG-1313-4.patch with additional Javadocs on PigServer and
PigServer.shutdown().
# triggerDeleteOnFail is still called by TestMultQueryLocal.executePlan, but
you are correct in that no one is (or was) calling registerDeleteOnFail, which
is the only entry point to push something onto the deleteOnFail stack.
I will gladly remove deleteOnFail and all calls to it as part of this JIRA, or
we can handle it in another one if that's cleaner w.r.t. issue tracking. Let me
know.
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch,
PIG-1313-1.patch, PIG-1313-2.patch, PIG-1313-3.patch, PIG-1313-4.patch,
Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated PIG-1313:
Component/s: impl
Affects Version/s: 0.7.0
Fix Version/s: 0.8.0
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.7.0
Reporter: Bill Graham
Assignee: Bill Graham
Fix For: 0.8.0
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch,
PIG-1313-1.patch, PIG-1313-2.patch, PIG-1313-3.patch, PIG-1313-4.patch,
Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Attachment: PIG-1313-0.4.0-4.patch
Here's the same patch for 0.4.0 if anyone wants it.
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.7.0
Reporter: Bill Graham
Assignee: Bill Graham
Fix For: 0.8.0
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-0.4.0-4.patch,
PIG-1313-1.patch, PIG-1313-1.patch, PIG-1313-2.patch, PIG-1313-3.patch,
PIG-1313-4.patch, Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Dai updated PIG-1313:
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
PIG-1313-4.patch committed to trunk. Will come with Pig 0.8 release. This issue
is about memory leak and it is hard to write a unit test for it. Bill tested it
manually and it works.
Thanks Bill for contributing!
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.7.0
Reporter: Bill Graham
Assignee: Bill Graham
Fix For: 0.8.0
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-0.4.0-4.patch,
PIG-1313-1.patch, PIG-1313-1.patch, PIG-1313-2.patch, PIG-1313-3.patch,
PIG-1313-4.patch, Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Status: Open (was: Patch Available)
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch,
PIG-1313-1.patch, PIG-1313-2.patch, Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Status: Patch Available (was: Open)
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch,
PIG-1313-1.patch, PIG-1313-2.patch, PIG-1313-3.patch, Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Attachment: PIG-1313-3.patch
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch,
PIG-1313-1.patch, PIG-1313-2.patch, PIG-1313-3.patch, Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Attachment: PIG-1313-0.4.0-1.patch
PIG-1313-1.patch
Attaching two patches, one built from the trunk and one built from the 0.4.0
branch (since 0.4.0 is what we're using). Not sure if patches to older branches
are still being applied or not though. Let me know if I should provide other
patches for other branches.
The patch includes the following changes to FileLocalizer:
- toDelete, deleteOnFail and relativeRoot are ThreadLocals.
- The initialized attribute and the init method has been removed, since they're
no longer needed.
In PigServer I've done the following:
- FileLocalizer.deleteTempFiles() is called from the shutdown method.
I've run my server with this patch applied for a few days now and the memory
consumption has stabilized. I also no longer see exceptions in the logs from
when a PigServers temp files would go missing.
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch,
Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Release Note: PigServer shutdown() method cleans up after itself better.
Status: Patch Available (was: Open)
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch,
Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Status: Patch Available (was: Open)
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch,
PIG-1313-1.patch, Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Attachment: PIG-1313-1.patch
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch,
PIG-1313-1.patch, Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Status: Open (was: Patch Available)
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch,
PIG-1313-1.patch, Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Status: Open (was: Patch Available)
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch,
PIG-1313-1.patch, PIG-1313-2.patch, Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Attachment: PIG-1313-2.patch
Previous patch caused issues with dependencies in test code, which I've
resolved in PIG-1313-2.patch. Also, the 0.4.0 patch was getting picked up and
was causing failures. Holding off on reattaching that patch until tests pass
with the trunk patch.
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch,
PIG-1313-1.patch, PIG-1313-2.patch, Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Status: Patch Available (was: Open)
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Assignee: Bill Graham
Attachments: PIG-1313-0.4.0-1.patch, PIG-1313-1.patch,
PIG-1313-1.patch, PIG-1313-2.patch, Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1313) PigServer leaks memory over time
[
https://issues.apache.org/jira/browse/PIG-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Graham updated PIG-1313:
-
Attachment: Pig1313Reproducer.java
Attaching {{Pig1313Reproducer.java}}, which is a class that can be used to
reproduce the issue. It re-runs a PigServer job in an infinite loop.
It's difficult to expose run time info about the toDelete Stack because it's
private but you can run a script like this to repeatedly dump the size/count of
the elements on the Stack:
{{while true; do jmap -histo pid | grep HFile; sleep 5; done}}
Over time the overall count of HFile objects will increase.
PigServer leaks memory over time
Key: PIG-1313
URL: https://issues.apache.org/jira/browse/PIG-1313
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
Attachments: Pig1313Reproducer.java
When {{PigServer}} runs it creates temporary files using the
{{FileLocalizer.getTemporaryPath(..)}}. This static method creates and
returns a handle to a temporary file (as an instance of
{{ElementDescriptor}}). The {{ElementDescriptors}} returned by this method
are kept on a static {{Stack}} named {{toDelete}}. The items on {{toDelete}}
get removed by the {{FileLocalizer.deleteTempFile()}} method.
The only place in the code where I see {{FileLocalizer.deleteTempFile()}}
called is in the Main class. {{PigServer}} does not call that method though,
so a long-running VM that repeatedly uses instances of {{PigServer}} to run
jobs will leak memory via {{toDelete}}.
One suggested fix is to have {{PigServer.shutdown()}} call
{{FileLocalizer.deleteTempFile()}}, but this would cause problems in a
multi-threaded environment, since it seems {{ElementDescriptors}} are pushed
onto the {{toDelete}} stack before they're used, not once they're done with.
With this approach, running multiple instances of {{PigServer}} in separate
threads could cause one completed job to clobber the other's still-in-use
temp files.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
