[jira] [Commented] (TEZ-4014) Allow DAGAppMaster to read configuration from plaintext (1/3)
[ https://issues.apache.org/jira/browse/TEZ-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811265#comment-16811265 ] Eric Wohlstadter commented on TEZ-4014: --- [~prasanth_j] [~jdere] [~gopalv] [~ashutoshc] Can you please help to review? This is (1/3) of patches for Standalone AM. The uber patch is attached to https://issues.apache.org/jira/browse/TEZ-3991. I'm splitting it up into 3 JIRAs to make it easier to review/manage. > Allow DAGAppMaster to read configuration from plaintext (1/3) > - > > Key: TEZ-4014 > URL: https://issues.apache.org/jira/browse/TEZ-4014 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-4014.1.patch, TEZ-4014.2.patch, TEZ-4014.3.patch, > TEZ-4014.4.patch, TEZ-4014.5.patch, TEZ-4014.6.patch > > > When launching DAGAppMaster independently, clients won't have to chance > localize the configuration for DAGAppMaster in the protobuf format. > In these cases it is more convenient for a platform to provide an AM > configuration as plaintext. > This Jira adds util methods to load tez-site.xml and > service_plugins_descriptor.json from the classpath. > Some changes are made to the plugin descriptor classes to make them > compatible with jackson. > These util methods are unused in this Jira (execpt by tests). In another > JIRA, they will get plugged into DAGAppMaster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4014) Allow DAGAppMaster to read configuration from plaintext (1/3)
[ https://issues.apache.org/jira/browse/TEZ-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-4014: -- Attachment: TEZ-4014.6.patch > Allow DAGAppMaster to read configuration from plaintext (1/3) > - > > Key: TEZ-4014 > URL: https://issues.apache.org/jira/browse/TEZ-4014 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-4014.1.patch, TEZ-4014.2.patch, TEZ-4014.3.patch, > TEZ-4014.4.patch, TEZ-4014.5.patch, TEZ-4014.6.patch > > > When launching DAGAppMaster independently, clients won't have to chance > localize the configuration for DAGAppMaster in the protobuf format. > In these cases it is more convenient for a platform to provide an AM > configuration as plaintext. > This Jira adds util methods to load tez-site.xml and > service_plugins_descriptor.json from the classpath. > Some changes are made to the plugin descriptor classes to make them > compatible with jackson. > These util methods are unused in this Jira (execpt by tests). In another > JIRA, they will get plugged into DAGAppMaster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3991) Unmanaged tez sessions
[ https://issues.apache.org/jira/browse/TEZ-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3991: -- Attachment: TEZ-3991.3.patch > Unmanaged tez sessions > -- > > Key: TEZ-3991 > URL: https://issues.apache.org/jira/browse/TEZ-3991 > Project: Apache Tez > Issue Type: New Feature >Affects Versions: 0.10.0 >Reporter: Prasanth Jayachandran >Assignee: Eric Wohlstadter >Priority: Major > Labels: Kubernetes > Attachments: TEZ-3991.1.patch, TEZ-3991.2.patch, TEZ-3991.3.patch > > > Provide an option for launching tez AM in unmanaged mode. In unmanaged mode, > tez AMs can register itself with Zookeeper which clients (like HiveServer2) > can discover via zk registry client. > HiveServer2 currently manages the lifecycle of tez AMs. The unmanaged mode > will let AM come up on their own (can be via simple java launcher) and be > discoverable for others. > Example use case for this is, HiveServer2 can discover already running AMs > and can attach to it for DAG submission and detach when done executing > queries. AMs can similarly discover LLAP daemons via task scheduler plugin > for submitting tasks. > A mode to cut off interactions with RM will also useful since for LLAP no > on-demand containers are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3991) Unmanaged tez sessions
[ https://issues.apache.org/jira/browse/TEZ-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3991: -- Attachment: TEZ-3991.2.patch > Unmanaged tez sessions > -- > > Key: TEZ-3991 > URL: https://issues.apache.org/jira/browse/TEZ-3991 > Project: Apache Tez > Issue Type: New Feature >Affects Versions: 0.10.0 >Reporter: Prasanth Jayachandran >Assignee: Eric Wohlstadter >Priority: Major > Labels: Kubernetes > Attachments: TEZ-3991.1.patch, TEZ-3991.2.patch > > > Provide an option for launching tez AM in unmanaged mode. In unmanaged mode, > tez AMs can register itself with Zookeeper which clients (like HiveServer2) > can discover via zk registry client. > HiveServer2 currently manages the lifecycle of tez AMs. The unmanaged mode > will let AM come up on their own (can be via simple java launcher) and be > discoverable for others. > Example use case for this is, HiveServer2 can discover already running AMs > and can attach to it for DAG submission and detach when done executing > queries. AMs can similarly discover LLAP daemons via task scheduler plugin > for submitting tasks. > A mode to cut off interactions with RM will also useful since for LLAP no > on-demand containers are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-4009) Zookeeper based AM Registry
[ https://issues.apache.org/jira/browse/TEZ-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter resolved TEZ-4009. --- Resolution: Duplicate Subsumed by TEZ-4007 > Zookeeper based AM Registry > --- > > Key: TEZ-4009 > URL: https://issues.apache.org/jira/browse/TEZ-4009 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-4009.1.patch > > > Zookeeper based implementation of the interface defined by TEZ-4008. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4007) Zookeeper based FrameworkServices and AmExtensions
[ https://issues.apache.org/jira/browse/TEZ-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-4007: -- Summary: Zookeeper based FrameworkServices and AmExtensions (was: Zookeeper based FrameworkClient) > Zookeeper based FrameworkServices and AmExtensions > -- > > Key: TEZ-4007 > URL: https://issues.apache.org/jira/browse/TEZ-4007 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > > A sub-class of FrameworkClient that interacts with application state managed > in Zookeeper rather than Yarn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4007) Zookeeper based FrameworkServices and AmExtensions (3/3)
[ https://issues.apache.org/jira/browse/TEZ-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-4007: -- Summary: Zookeeper based FrameworkServices and AmExtensions (3/3) (was: Zookeeper based FrameworkServices and AmExtensions) > Zookeeper based FrameworkServices and AmExtensions (3/3) > > > Key: TEZ-4007 > URL: https://issues.apache.org/jira/browse/TEZ-4007 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > > A sub-class of FrameworkClient that interacts with application state managed > in Zookeeper rather than Yarn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4008) Pluggable AM FrameworkServices and AmExtensions (2/3)
[
https://issues.apache.org/jira/browse/TEZ-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-4008:
--
Summary: Pluggable AM FrameworkServices and AmExtensions (2/3) (was:
Pluggable AM FrameworkServices and AmExtensions)
> Pluggable AM FrameworkServices and AmExtensions (2/3)
> -
>
> Key: TEZ-4008
> URL: https://issues.apache.org/jira/browse/TEZ-4008
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-4008.1.patch, TEZ-4008.2.patch, TEZ-4008.3.patch,
> TEZ-4008.4.patch
>
>
> Interface and reflective plugin configuration for a pool of AMs identified by
> a namespace.
> The registry should allow each {{DAGClientServer}} to register/unregister
> themselves from a pool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4008) Pluggable AM FrameworkServices and AmExtensions
[
https://issues.apache.org/jira/browse/TEZ-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-4008:
--
Summary: Pluggable AM FrameworkServices and AmExtensions (was: Pluggable
AM pool registry)
> Pluggable AM FrameworkServices and AmExtensions
> ---
>
> Key: TEZ-4008
> URL: https://issues.apache.org/jira/browse/TEZ-4008
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-4008.1.patch, TEZ-4008.2.patch, TEZ-4008.3.patch,
> TEZ-4008.4.patch
>
>
> Interface and reflective plugin configuration for a pool of AMs identified by
> a namespace.
> The registry should allow each {{DAGClientServer}} to register/unregister
> themselves from a pool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4014) Allow DAGAppMaster to read configuration from plaintext (1/3)
[ https://issues.apache.org/jira/browse/TEZ-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-4014: -- Attachment: TEZ-4014.5.patch > Allow DAGAppMaster to read configuration from plaintext (1/3) > - > > Key: TEZ-4014 > URL: https://issues.apache.org/jira/browse/TEZ-4014 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-4014.1.patch, TEZ-4014.2.patch, TEZ-4014.3.patch, > TEZ-4014.4.patch, TEZ-4014.5.patch > > > When launching DAGAppMaster independently, clients won't have to chance > localize the configuration for DAGAppMaster in the protobuf format. > In these cases it is more convenient for a platform to provide an AM > configuration as plaintext. > This Jira adds util methods to load tez-site.xml and > service_plugins_descriptor.json from the classpath. > Some changes are made to the plugin descriptor classes to make them > compatible with jackson. > These util methods are unused in this Jira (execpt by tests). In another > JIRA, they will get plugged into DAGAppMaster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4014) Allow DAGAppMaster to read configuration from plaintext (1/3)
[ https://issues.apache.org/jira/browse/TEZ-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-4014: -- Summary: Allow DAGAppMaster to read configuration from plaintext (1/3) (was: Allow DAGAppMaster to read configuration from plaintext) > Allow DAGAppMaster to read configuration from plaintext (1/3) > - > > Key: TEZ-4014 > URL: https://issues.apache.org/jira/browse/TEZ-4014 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-4014.1.patch, TEZ-4014.2.patch, TEZ-4014.3.patch, > TEZ-4014.4.patch > > > When launching DAGAppMaster independently, clients won't have to chance > localize the configuration for DAGAppMaster in the protobuf format. > In these cases it is more convenient for a platform to provide an AM > configuration as plaintext. > This Jira adds util methods to load tez-site.xml and > service_plugins_descriptor.json from the classpath. > Some changes are made to the plugin descriptor classes to make them > compatible with jackson. > These util methods are unused in this Jira (execpt by tests). In another > JIRA, they will get plugged into DAGAppMaster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3991) Unmanaged tez sessions
[ https://issues.apache.org/jira/browse/TEZ-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3991: -- Attachment: TEZ-3991.1.patch > Unmanaged tez sessions > -- > > Key: TEZ-3991 > URL: https://issues.apache.org/jira/browse/TEZ-3991 > Project: Apache Tez > Issue Type: New Feature >Affects Versions: 0.10.0 >Reporter: Prasanth Jayachandran >Assignee: Eric Wohlstadter >Priority: Major > Labels: Kubernetes > Attachments: TEZ-3991.1.patch > > > Provide an option for launching tez AM in unmanaged mode. In unmanaged mode, > tez AMs can register itself with Zookeeper which clients (like HiveServer2) > can discover via zk registry client. > HiveServer2 currently manages the lifecycle of tez AMs. The unmanaged mode > will let AM come up on their own (can be via simple java launcher) and be > discoverable for others. > Example use case for this is, HiveServer2 can discover already running AMs > and can attach to it for DAG submission and detach when done executing > queries. AMs can similarly discover LLAP daemons via task scheduler plugin > for submitting tasks. > A mode to cut off interactions with RM will also useful since for LLAP no > on-demand containers are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4057) Fix Unsorted broadcast shuffle umasks
[
https://issues.apache.org/jira/browse/TEZ-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-4057:
--
Attachment: TEZ-4057.1.patch
> Fix Unsorted broadcast shuffle umasks
> -
>
> Key: TEZ-4057
> URL: https://issues.apache.org/jira/browse/TEZ-4057
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.2
>Reporter: Gopal V
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-4057.1.patch
>
>
> {code}
> if (numPartitions == 1 && !pipelinedShuffle) {
> //special case, where in only one partition is available.
> finalOutPath = outputFileHandler.getOutputFileForWrite();
> finalIndexPath =
> outputFileHandler.getOutputIndexFileForWrite(indexFileSizeEstimate);
> skipBuffers = true;
> writer = new IFile.Writer(conf, rfs, finalOutPath, keyClass, valClass,
> codec, outputRecordsCounter, outputRecordBytesCounter);
> } else {
> skipBuffers = false;
> writer = null;
> }
> {code}
> The broadcast events don't update the file umasks, because they have 1
> partition.
> {code}
> total 8.0K
> -rw--- 1 hive hadoop 15 Mar 27 20:30 file.out
> -rw-r- 1 hive hadoop 32 Mar 27 20:30 file.out.index
> {code}
> ending up with readable index files and unreadable .out files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (TEZ-4057) Fix Unsorted broadcast shuffle umasks
[
https://issues.apache.org/jira/browse/TEZ-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter reassigned TEZ-4057:
-
Assignee: Eric Wohlstadter
> Fix Unsorted broadcast shuffle umasks
> -
>
> Key: TEZ-4057
> URL: https://issues.apache.org/jira/browse/TEZ-4057
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.2
>Reporter: Gopal V
>Assignee: Eric Wohlstadter
>Priority: Major
>
> {code}
> if (numPartitions == 1 && !pipelinedShuffle) {
> //special case, where in only one partition is available.
> finalOutPath = outputFileHandler.getOutputFileForWrite();
> finalIndexPath =
> outputFileHandler.getOutputIndexFileForWrite(indexFileSizeEstimate);
> skipBuffers = true;
> writer = new IFile.Writer(conf, rfs, finalOutPath, keyClass, valClass,
> codec, outputRecordsCounter, outputRecordBytesCounter);
> } else {
> skipBuffers = false;
> writer = null;
> }
> {code}
> The broadcast events don't update the file umasks, because they have 1
> partition.
> {code}
> total 8.0K
> -rw--- 1 hive hadoop 15 Mar 27 20:30 file.out
> -rw-r- 1 hive hadoop 32 Mar 27 20:30 file.out.index
> {code}
> ending up with readable index files and unreadable .out files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4028) Events not visible from proto history logging for s3a filesystem until dag completes.
[ https://issues.apache.org/jira/browse/TEZ-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740590#comment-16740590 ] Eric Wohlstadter commented on TEZ-4028: --- [~harishjp] lgtm [~sseth], can you merge this? It makes the ProtoBuf based logger work in the cloud (since cloud blob stores don't support streaming file append). > Events not visible from proto history logging for s3a filesystem until dag > completes. > - > > Key: TEZ-4028 > URL: https://issues.apache.org/jira/browse/TEZ-4028 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: TEZ-4028.01.patch, TEZ-4028.02.patch > > > The events are not visible in the files because s3 filesystem > * flush writes to local disk and only upload/commit to s3 on close. > * does not support append > As an initial fix we log the dag submitted, initialized and started events > into a file and these can be read to get the dag plan, config from the AM. > The counters are anyways not available until the dag completes. > The in-progress information cannot be read, this can be obtained from the AM > once we have the above events. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4014) Allow DAGAppMaster to read configuration from plaintext
[ https://issues.apache.org/jira/browse/TEZ-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-4014: -- Description: When launching DAGAppMaster independently, clients won't have to chance localize the configuration for DAGAppMaster in the protobuf format. In these cases it is more convenient for a platform to provide an AM configuration as plaintext. This Jira adds util methods to load tez-site.xml and service_plugins_descriptor.json from the classpath. Some changes are made to the plugin descriptor classes to make them compatible with jackson. These util methods are unused in this Jira (execpt by tests). In another JIRA, they will get plugged into DAGAppMaster. was: When launching DAGAppMaster independently, clients won't have to chance localize the configuration for DAGAppMaster in the protobuf format. In these cases it is more convenient for a platform to provide an AM configuration as plaintext. The change here will be to look for the protobuf conf first, and then fall back to: # Looking for a tez-site.xml text file in the classpath # Looking for a service_plugins_descriptor in JSON format. > Allow DAGAppMaster to read configuration from plaintext > --- > > Key: TEZ-4014 > URL: https://issues.apache.org/jira/browse/TEZ-4014 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-4014.1.patch, TEZ-4014.2.patch, TEZ-4014.3.patch, > TEZ-4014.4.patch > > > When launching DAGAppMaster independently, clients won't have to chance > localize the configuration for DAGAppMaster in the protobuf format. > In these cases it is more convenient for a platform to provide an AM > configuration as plaintext. > This Jira adds util methods to load tez-site.xml and > service_plugins_descriptor.json from the classpath. > Some changes are made to the plugin descriptor classes to make them > compatible with jackson. > These util methods are unused in this Jira (execpt by tests). In another > JIRA, they will get plugged into DAGAppMaster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4014) Allow DAGAppMaster to read configuration from plaintext
[ https://issues.apache.org/jira/browse/TEZ-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-4014: -- Attachment: TEZ-4014.4.patch > Allow DAGAppMaster to read configuration from plaintext > --- > > Key: TEZ-4014 > URL: https://issues.apache.org/jira/browse/TEZ-4014 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-4014.1.patch, TEZ-4014.2.patch, TEZ-4014.3.patch, > TEZ-4014.4.patch > > > When launching DAGAppMaster independently, clients won't have to chance > localize the configuration for DAGAppMaster in the protobuf format. > In these cases it is more convenient for a platform to provide an AM > configuration as plaintext. > The change here will be to look for the protobuf conf first, and then fall > back to: > # Looking for a tez-site.xml text file in the classpath > # Looking for a service_plugins_descriptor in JSON format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4014) Allow DAGAppMaster to read configuration from plaintext
[ https://issues.apache.org/jira/browse/TEZ-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-4014: -- Attachment: TEZ-4014.3.patch > Allow DAGAppMaster to read configuration from plaintext > --- > > Key: TEZ-4014 > URL: https://issues.apache.org/jira/browse/TEZ-4014 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-4014.1.patch, TEZ-4014.2.patch, TEZ-4014.3.patch > > > When launching DAGAppMaster independently, clients won't have to chance > localize the configuration for DAGAppMaster in the protobuf format. > In these cases it is more convenient for a platform to provide an AM > configuration as plaintext. > The change here will be to look for the protobuf conf first, and then fall > back to: > # Looking for a tez-site.xml text file in the classpath > # Looking for a service_plugins_descriptor in JSON format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4014) Allow DAGAppMaster to read configuration from plaintext
[ https://issues.apache.org/jira/browse/TEZ-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-4014: -- Attachment: TEZ-4014.2.patch > Allow DAGAppMaster to read configuration from plaintext > --- > > Key: TEZ-4014 > URL: https://issues.apache.org/jira/browse/TEZ-4014 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-4014.1.patch, TEZ-4014.2.patch > > > When launching DAGAppMaster independently, clients won't have to chance > localize the configuration for DAGAppMaster in the protobuf format. > In these cases it is more convenient for a platform to provide an AM > configuration as plaintext. > The change here will be to look for the protobuf conf first, and then fall > back to: > # Looking for a tez-site.xml text file in the classpath > # Looking for a service_plugins_descriptor in JSON format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4028) Events not visible from proto history logging for s3a filesystem until dag completes.
[ https://issues.apache.org/jira/browse/TEZ-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736408#comment-16736408 ] Eric Wohlstadter commented on TEZ-4028: --- [~harishjp] I was initially confused why "_1" was being added. Minor: Could you define that string as a constant with a descriptive name to indicate it is related to TEZ_HISTORY_LOGGING_PROTO_SPLIT_DAG_START? > Events not visible from proto history logging for s3a filesystem until dag > completes. > - > > Key: TEZ-4028 > URL: https://issues.apache.org/jira/browse/TEZ-4028 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: TEZ-4028.01.patch > > > The events are not visible in the files because s3 filesystem > * flush writes to local disk and only upload/commit to s3 on close. > * does not support append > As an initial fix we log the dag submitted, initialized and started events > into a file and these can be read to get the dag plan, config from the AM. > The counters are anyways not available until the dag completes. > The in-progress information cannot be read, this can be obtained from the AM > once we have the above events. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-3957:
--
Attachment: TEZ-3957.03.patch
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3957.01.patch, TEZ-3957.02.patch, TEZ-3957.02.patch,
> TEZ-3957.03.patch, TEZ-3957.patch
>
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-4024) DAGAppMaster should access resources in UGI context on startup
Eric Wohlstadter created TEZ-4024: - Summary: DAGAppMaster should access resources in UGI context on startup Key: TEZ-4024 URL: https://issues.apache.org/jira/browse/TEZ-4024 Project: Apache Tez Issue Type: Bug Affects Versions: 0.9.1 Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter On startup, DAGAppMaster accesses resources such as config PBs and staging dirs as yarn rather than through the UGI context. This creates access control violations when yarn does not have permissions for those resources. This also seems odd, since during shutdown, DAGAppMaster does access resources through UGI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4006) Make FrameworkClient Pluggable
[
https://issues.apache.org/jira/browse/TEZ-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688691#comment-16688691
]
Eric Wohlstadter commented on TEZ-4006:
---
[~hitesh]
Yeah, I think we need to generalize the concept of "modes" beyond local and
session.
While working on this umbrella, I'm seeing a lot of "families" of configuration
options, plugins, etc.. that need to be specified as a group, for things to
work properly. Better to encapsulate them somehow, I'll look into service
loader.
> Make FrameworkClient Pluggable
> --
>
> Key: TEZ-4006
> URL: https://issues.apache.org/jira/browse/TEZ-4006
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-4006.1.patch
>
>
> Currently {{FrameworkClient.createFrameworkClient}} is hardcoded to use
> either LocalClient or YarnClient.
> Change it to allow specifying a FrameworkClient impl. class from
> configuration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-2222) Investigate moving to log4j2 for logging
[ https://issues.apache.org/jira/browse/TEZ-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681989#comment-16681989 ] Eric Wohlstadter commented on TEZ-: --- [~yingdachen] I realize this is off-topic for this particular JIRA, but since you mentioned it: We are hoping to have something like you described here: "resource/runtime abstraction layer that would allow Tez-as-a-library to run on different distributed framework, including YARN/Mesos/K8S etc". We have started work on this here: https://issues.apache.org/jira/browse/TEZ-3991 Do you have a patch that you can share at this time? Or any idea when you might have one available? I'm trying to see if we can avoid overlapping work, and/or maybe we could start collaborating on this upstream. > Investigate moving to log4j2 for logging > > > Key: TEZ- > URL: https://issues.apache.org/jira/browse/TEZ- > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Priority: Blocker > > Via slf4j. > Some bits to keep in mind > - We have explicit code which rotates logs using direct log4j12 APIs. This > should keep working. I believe the log4j2 APIs are different here > - API compatibility between log4j12 / log4j2 can be problematic - if both end > up on the classpath (I believe the APIs are different) > - Hadoop dist includes a slf4j-log4j12 binding. Changing the default can > result in sl4j-log4j12 and slf4j-log4j2 to co-exist by default - which could > be problematic. Needs investigation. > End of the day, we will likely need an option to use either of the two. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4008) Pluggable AM pool registry
[
https://issues.apache.org/jira/browse/TEZ-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-4008:
--
Attachment: TEZ-4008.4.patch
> Pluggable AM pool registry
> --
>
> Key: TEZ-4008
> URL: https://issues.apache.org/jira/browse/TEZ-4008
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-4008.1.patch, TEZ-4008.2.patch, TEZ-4008.3.patch,
> TEZ-4008.4.patch
>
>
> Interface and reflective plugin configuration for a pool of AMs identified by
> a namespace.
> The registry should allow each {{DAGClientServer}} to register/unregister
> themselves from a pool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4008) Pluggable AM pool registry
[
https://issues.apache.org/jira/browse/TEZ-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-4008:
--
Attachment: TEZ-4008.3.patch
> Pluggable AM pool registry
> --
>
> Key: TEZ-4008
> URL: https://issues.apache.org/jira/browse/TEZ-4008
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-4008.1.patch, TEZ-4008.2.patch, TEZ-4008.3.patch
>
>
> Interface and reflective plugin configuration for a pool of AMs identified by
> a namespace.
> The registry should allow each {{DAGClientServer}} to register/unregister
> themselves from a pool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4016) Make TEZ_AM_LOCAL_RESOURCES_PB optional for unmanaged AM
[
https://issues.apache.org/jira/browse/TEZ-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-4016:
--
Attachment: TEZ-4016.1.patch
> Make TEZ_AM_LOCAL_RESOURCES_PB optional for unmanaged AM
>
>
> Key: TEZ-4016
> URL: https://issues.apache.org/jira/browse/TEZ-4016
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-4016.1.patch
>
>
> In session mode, {{DAGAppMaster}} requires the
> {{tez.session.local-resources.pb}} file to be present.
> If resources are localized by a framework other than YARN, this may not be
> required.
> Make it optional.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-4016) Make TEZ_AM_LOCAL_RESOURCES_PB optional for unmanaged AM
Eric Wohlstadter created TEZ-4016:
-
Summary: Make TEZ_AM_LOCAL_RESOURCES_PB optional for unmanaged AM
Key: TEZ-4016
URL: https://issues.apache.org/jira/browse/TEZ-4016
Project: Apache Tez
Issue Type: Sub-task
Reporter: Eric Wohlstadter
Assignee: Eric Wohlstadter
In session mode, {{DAGAppMaster}} requires the
{{tez.session.local-resources.pb}} file to be present.
If resources are localized by a framework other than YARN, this may not be
required.
Make it optional.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4014) Allow DAGAppMaster to read configuration from plaintext
[ https://issues.apache.org/jira/browse/TEZ-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-4014: -- Attachment: TEZ-4014.1.patch > Allow DAGAppMaster to read configuration from plaintext > --- > > Key: TEZ-4014 > URL: https://issues.apache.org/jira/browse/TEZ-4014 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-4014.1.patch > > > When launching DAGAppMaster independently, clients won't have to chance > localize the configuration for DAGAppMaster in the protobuf format. > In these cases it is more convenient for a platform to provide an AM > configuration as plaintext. > The change here will be to look for the protobuf conf first, and then fall > back to: > # Looking for a tez-site.xml text file in the classpath > # Looking for a service_plugins_descriptor in JSON format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4014) Allow DAGAppMaster to read configuration from plaintext
[ https://issues.apache.org/jira/browse/TEZ-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-4014: -- Description: When launching DAGAppMaster independently, clients won't have to chance localize the configuration for DAGAppMaster in the protobuf format. In these cases it is more convenient for a platform to provide an AM configuration as plaintext. The change here will be to look for the protobuf conf first, and then fall back to: # Looking for a tez-site.xml text file in the classpath # Looking for a service_plugins_descriptor in JSON format. was: When launching DAGAppMaster independently, clients won't have to chance localize the configuration for DAGAppMaster in the protobuf format. In these cases it is more convenient for a platform to provide an AM configuration as a plaintext tez-site.xml. The change here will be to look for the protobuf conf first, and then fall back to looking for a tez-site.xml text file in the classpath. > Allow DAGAppMaster to read configuration from plaintext > --- > > Key: TEZ-4014 > URL: https://issues.apache.org/jira/browse/TEZ-4014 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > > When launching DAGAppMaster independently, clients won't have to chance > localize the configuration for DAGAppMaster in the protobuf format. > In these cases it is more convenient for a platform to provide an AM > configuration as plaintext. > The change here will be to look for the protobuf conf first, and then fall > back to: > # Looking for a tez-site.xml text file in the classpath > # Looking for a service_plugins_descriptor in JSON format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4014) Allow DAGAppMaster to read configuration from plaintext
[ https://issues.apache.org/jira/browse/TEZ-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-4014: -- Summary: Allow DAGAppMaster to read configuration from plaintext (was: Allow DAGAppMaster to read configuration from tez-site.xml plaintext) > Allow DAGAppMaster to read configuration from plaintext > --- > > Key: TEZ-4014 > URL: https://issues.apache.org/jira/browse/TEZ-4014 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > > When launching DAGAppMaster independently, clients won't have to chance > localize the configuration for DAGAppMaster in the protobuf format. > In these cases it is more convenient for a platform to provide an AM > configuration as a plaintext tez-site.xml. > The change here will be to look for the protobuf conf first, and then fall > back to looking for a tez-site.xml text file in the classpath. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669397#comment-16669397
]
Eric Wohlstadter commented on TEZ-3957:
---
lgtm (unbinding)
[~jeagles]
I'm pretty sure MR doesn't have this counter (at least as of 4 years ago).
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3957.01.patch, TEZ-3957.patch
>
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4006) Make FrameworkClient Pluggable
[
https://issues.apache.org/jira/browse/TEZ-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668069#comment-16668069
]
Eric Wohlstadter commented on TEZ-4006:
---
[~gopalv] [~sseth] [~jeagles] [~jlowe]
Could someone help to review this one?
It is self contained and should be generally useful even outside the context of
this umbrella jira.
> Make FrameworkClient Pluggable
> --
>
> Key: TEZ-4006
> URL: https://issues.apache.org/jira/browse/TEZ-4006
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-4006.1.patch
>
>
> Currently {{FrameworkClient.createFrameworkClient}} is hardcoded to use
> either LocalClient or YarnClient.
> Change it to allow specifying a FrameworkClient impl. class from
> configuration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-4014) Allow DAGAppMaster to read configuration from tez-site.xml plaintext
Eric Wohlstadter created TEZ-4014: - Summary: Allow DAGAppMaster to read configuration from tez-site.xml plaintext Key: TEZ-4014 URL: https://issues.apache.org/jira/browse/TEZ-4014 Project: Apache Tez Issue Type: Sub-task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter When launching DAGAppMaster independently, clients won't have to chance localize the configuration for DAGAppMaster in the protobuf format. In these cases it is more convenient for a platform to provide an AM configuration as a plaintext tez-site.xml. The change here will be to look for the protobuf conf first, and then fall back to looking for a tez-site.xml text file in the classpath. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664655#comment-16664655
]
Eric Wohlstadter commented on TEZ-3957:
---
[~sershe]
Do you want the counter to be continuously updated and reported back to the AM
from running task containers?
I was planning to just add the counter at DAG completion time, from information
in the AM about completed tasks. That's what I meant by "Report
TASK_DURATION_MILLIS as a Counter for completed tasks".
What I had in mind is less useful in general than what I understand this patch
is doing, but it doesn't add any overhead to AM <-> Container messaging.
Wanted to check that there is a use-case where having this as a running counter
is useful (it wasn't necessary for my use-case).
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: TEZ-3957.patch
>
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4009) Zookeeper based AM Registry
[ https://issues.apache.org/jira/browse/TEZ-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657551#comment-16657551 ] Eric Wohlstadter commented on TEZ-4009: --- Depends on TEZ-4008. > Zookeeper based AM Registry > --- > > Key: TEZ-4009 > URL: https://issues.apache.org/jira/browse/TEZ-4009 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-4009.1.patch > > > Zookeeper based implementation of the interface defined by TEZ-4008. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4009) Zookeeper based AM Registry
[ https://issues.apache.org/jira/browse/TEZ-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-4009: -- Attachment: TEZ-4009.1.patch > Zookeeper based AM Registry > --- > > Key: TEZ-4009 > URL: https://issues.apache.org/jira/browse/TEZ-4009 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-4009.1.patch > > > Zookeeper based implementation of the interface defined by TEZ-4008. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4008) Pluggable AM pool registry
[
https://issues.apache.org/jira/browse/TEZ-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-4008:
--
Attachment: TEZ-4008.2.patch
> Pluggable AM pool registry
> --
>
> Key: TEZ-4008
> URL: https://issues.apache.org/jira/browse/TEZ-4008
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-4008.1.patch, TEZ-4008.2.patch
>
>
> Interface and reflective plugin configuration for a pool of AMs identified by
> a namespace.
> The registry should allow each {{DAGClientServer}} to register/unregister
> themselves from a pool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4008) Pluggable AM pool registry
[
https://issues.apache.org/jira/browse/TEZ-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-4008:
--
Attachment: TEZ-4008.1.patch
> Pluggable AM pool registry
> --
>
> Key: TEZ-4008
> URL: https://issues.apache.org/jira/browse/TEZ-4008
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-4008.1.patch
>
>
> Interface and reflective plugin configuration for a pool of AMs identified by
> a namespace.
> The registry should allow each {{DAGClientServer}} to register/unregister
> themselves from a pool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4006) Make FrameworkClient Pluggable
[
https://issues.apache.org/jira/browse/TEZ-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654160#comment-16654160
]
Eric Wohlstadter commented on TEZ-4006:
---
[~jeagles]
Thanks!
> Make FrameworkClient Pluggable
> --
>
> Key: TEZ-4006
> URL: https://issues.apache.org/jira/browse/TEZ-4006
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-4006.1.patch
>
>
> Currently {{FrameworkClient.createFrameworkClient}} is hardcoded to use
> either LocalClient or YarnClient.
> Change it to allow specifying a FrameworkClient impl. class from
> configuration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4006) Make FrameworkClient Pluggable
[
https://issues.apache.org/jira/browse/TEZ-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-4006:
--
Attachment: TEZ-4006.1.patch
> Make FrameworkClient Pluggable
> --
>
> Key: TEZ-4006
> URL: https://issues.apache.org/jira/browse/TEZ-4006
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-4006.1.patch
>
>
> Currently {{FrameworkClient.createFrameworkClient}} is hardcoded to use
> either LocalClient or YarnClient.
> Change it to allow specifying a FrameworkClient impl. class from
> configuration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3991) Unmanaged tez sessions
[ https://issues.apache.org/jira/browse/TEZ-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3991: -- Target Version/s: 0.10.1 (was: 0.10.0) > Unmanaged tez sessions > -- > > Key: TEZ-3991 > URL: https://issues.apache.org/jira/browse/TEZ-3991 > Project: Apache Tez > Issue Type: New Feature >Affects Versions: 0.10.0 >Reporter: Prasanth Jayachandran >Assignee: Eric Wohlstadter >Priority: Major > > Provide an option for launching tez AM in unmanaged mode. In unmanaged mode, > tez AMs can register itself with Zookeeper which clients (like HiveServer2) > can discover via zk registry client. > HiveServer2 currently manages the lifecycle of tez AMs. The unmanaged mode > will let AM come up on their own (can be via simple java launcher) and be > discoverable for others. > Example use case for this is, HiveServer2 can discover already running AMs > and can attach to it for DAG submission and detach when done executing > queries. AMs can similarly discover LLAP daemons via task scheduler plugin > for submitting tasks. > A mode to cut off interactions with RM will also useful since for LLAP no > on-demand containers are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-4009) Zookeeper based AM Registry
Eric Wohlstadter created TEZ-4009: - Summary: Zookeeper based AM Registry Key: TEZ-4009 URL: https://issues.apache.org/jira/browse/TEZ-4009 Project: Apache Tez Issue Type: Sub-task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter Zookeeper based implementation of the interface defined by TEZ-4008. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-4008) Pluggable AM pool registry
Eric Wohlstadter created TEZ-4008:
-
Summary: Pluggable AM pool registry
Key: TEZ-4008
URL: https://issues.apache.org/jira/browse/TEZ-4008
Project: Apache Tez
Issue Type: Sub-task
Reporter: Eric Wohlstadter
Assignee: Eric Wohlstadter
Interface and reflective plugin configuration for a pool of AMs identified by a
namespace.
The registry should allow each {{DAGClientServer}} to register/unregister
themselves from a pool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-4007) Zookeeper based FrameworkClient
Eric Wohlstadter created TEZ-4007: - Summary: Zookeeper based FrameworkClient Key: TEZ-4007 URL: https://issues.apache.org/jira/browse/TEZ-4007 Project: Apache Tez Issue Type: Sub-task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter A sub-class of FrameworkClient that interacts with application state managed in Zookeeper rather than Yarn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-4006) Make FrameworkClient Pluggable
Eric Wohlstadter created TEZ-4006:
-
Summary: Make FrameworkClient Pluggable
Key: TEZ-4006
URL: https://issues.apache.org/jira/browse/TEZ-4006
Project: Apache Tez
Issue Type: Sub-task
Reporter: Eric Wohlstadter
Assignee: Eric Wohlstadter
Currently {{FrameworkClient.createFrameworkClient}} is hardcoded to use either
LocalClient or YarnClient.
Change it to allow specifying a FrameworkClient impl. class from configuration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (TEZ-3991) Unmanaged tez sessions
[ https://issues.apache.org/jira/browse/TEZ-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter reassigned TEZ-3991: - Assignee: Eric Wohlstadter > Unmanaged tez sessions > -- > > Key: TEZ-3991 > URL: https://issues.apache.org/jira/browse/TEZ-3991 > Project: Apache Tez > Issue Type: New Feature >Affects Versions: 0.10.0 >Reporter: Prasanth Jayachandran >Assignee: Eric Wohlstadter >Priority: Major > > Provide an option for launching tez AM in unmanaged mode. In unmanaged mode, > tez AMs can register itself with Zookeeper which clients (like HiveServer2) > can discover via zk registry client. > HiveServer2 currently manages the lifecycle of tez AMs. The unmanaged mode > will let AM come up on their own (can be via simple java launcher) and be > discoverable for others. > Example use case for this is, HiveServer2 can discover already running AMs > and can attach to it for DAG submission and detach when done executing > queries. AMs can similarly discover LLAP daemons via task scheduler plugin > for submitting tasks. > A mode to cut off interactions with RM will also useful since for LLAP no > on-demand containers are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3986) add CHANGES.txt (branch-0.10.0 only)
[ https://issues.apache.org/jira/browse/TEZ-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3986: -- Attachment: TEZ-3986.3.patch > add CHANGES.txt (branch-0.10.0 only) > > > Key: TEZ-3986 > URL: https://issues.apache.org/jira/browse/TEZ-3986 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3986.1.patch, TEZ-3986.2.patch, TEZ-3986.3.patch > > > Commit only to branch-0.10.0. > Do not commit to master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3986) add CHANGES.txt (branch-0.10.0 only)
[ https://issues.apache.org/jira/browse/TEZ-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620950#comment-16620950 ] Eric Wohlstadter commented on TEZ-3986: --- Updated patch to include only CHANGES.txt. Transforming pom.xml's from 0.10.0-SNAPSHOT to 0.10.0 is taken care of by the Maven Versions Plugin. > add CHANGES.txt (branch-0.10.0 only) > > > Key: TEZ-3986 > URL: https://issues.apache.org/jira/browse/TEZ-3986 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3986.1.patch, TEZ-3986.2.patch, TEZ-3986.3.patch > > > Commit only to branch-0.10.0. > Do not commit to master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3986) add CHANGES.txt (branch-0.10.0 only)
[ https://issues.apache.org/jira/browse/TEZ-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3986: -- Summary: add CHANGES.txt (branch-0.10.0 only) (was: Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only)) > add CHANGES.txt (branch-0.10.0 only) > > > Key: TEZ-3986 > URL: https://issues.apache.org/jira/browse/TEZ-3986 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3986.1.patch, TEZ-3986.2.patch > > > Commit only to branch-0.10.0. > Do not commit to master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (TEZ-3986) Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only)
[ https://issues.apache.org/jira/browse/TEZ-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614126#comment-16614126 ] Eric Wohlstadter edited comment on TEZ-3986 at 9/13/18 10:50 PM: - [~jeagles] [~jlowe] -Can you review/commit this change to branch-0.10.0?- -Then I can build and sign the bin and src artifacts, and make them available for upload to Apache and Nexus staging.- Oops, nevermind this. It looks like the artifacts deployed to Nexus staging must be a SNAPSHOT version, and they will be renamed by the deploy process: [http://www.apache.org/dev/publishing-maven-artifacts.html#prepare-poms] was (Author: ewohlstadter): [~jeagles] [~jlowe] Can you review/commit this change to branch-0.10.0? Then I can build and sign the bin and src artifacts, and make them available for upload to Apache and Nexus staging. > Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only) > > > Key: TEZ-3986 > URL: https://issues.apache.org/jira/browse/TEZ-3986 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3986.1.patch, TEZ-3986.2.patch > > > Commit only to branch-0.10.0. > Do not commit to master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3988) Update snapshot version in master to 0.10.1-SNAPSHOT
[ https://issues.apache.org/jira/browse/TEZ-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614127#comment-16614127 ] Eric Wohlstadter commented on TEZ-3988: --- [~jeagles] [~jlowe] Can you go ahead and review/commit this to master? > Update snapshot version in master to 0.10.1-SNAPSHOT > > > Key: TEZ-3988 > URL: https://issues.apache.org/jira/browse/TEZ-3988 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3988.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3986) Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only)
[ https://issues.apache.org/jira/browse/TEZ-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16614126#comment-16614126 ] Eric Wohlstadter commented on TEZ-3986: --- [~jeagles] [~jlowe] Can you review/commit this change to branch-0.10.0? Then I can build and sign the bin and src artifacts, and make them available for upload to Apache and Nexus staging. > Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only) > > > Key: TEZ-3986 > URL: https://issues.apache.org/jira/browse/TEZ-3986 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3986.1.patch, TEZ-3986.2.patch > > > Commit only to branch-0.10.0. > Do not commit to master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3986) Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only)
[ https://issues.apache.org/jira/browse/TEZ-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3986: -- Attachment: TEZ-3986.2.patch > Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only) > > > Key: TEZ-3986 > URL: https://issues.apache.org/jira/browse/TEZ-3986 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3986.1.patch, TEZ-3986.2.patch > > > Commit only to branch-0.10.0. > Do not commit to master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3986) Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only)
[ https://issues.apache.org/jira/browse/TEZ-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606269#comment-16606269 ] Eric Wohlstadter commented on TEZ-3986: --- [~jeagles] Ok, I'm seeing the new branch-0.10.0 in github now, so I'll go ahead and complete this JIRA. > Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only) > > > Key: TEZ-3986 > URL: https://issues.apache.org/jira/browse/TEZ-3986 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3986.1.patch > > > Commit only to branch-0.10.0. > Do not commit to master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3986) Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only)
[ https://issues.apache.org/jira/browse/TEZ-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605055#comment-16605055 ] Eric Wohlstadter commented on TEZ-3986: --- [~jeagles] I'm still seeing the 0.9.2 branch for branch-0.10.0 [https://github.com/apache/tez/blob/branch-0.10.0/pom.xml] Can you check again? > Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only) > > > Key: TEZ-3986 > URL: https://issues.apache.org/jira/browse/TEZ-3986 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3986.1.patch > > > Commit only to branch-0.10.0. > Do not commit to master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3988) Update snapshot version in master to 0.10.1-SNAPSHOT
[ https://issues.apache.org/jira/browse/TEZ-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3988: -- Attachment: TEZ-3988.1.patch > Update snapshot version in master to 0.10.1-SNAPSHOT > > > Key: TEZ-3988 > URL: https://issues.apache.org/jira/browse/TEZ-3988 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3988.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3988) Update snapshot version in master to 0.10.1-SNAPSHOT
Eric Wohlstadter created TEZ-3988: - Summary: Update snapshot version in master to 0.10.1-SNAPSHOT Key: TEZ-3988 URL: https://issues.apache.org/jira/browse/TEZ-3988 Project: Apache Tez Issue Type: Task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3986) Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only)
[ https://issues.apache.org/jira/browse/TEZ-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604823#comment-16604823 ] Eric Wohlstadter commented on TEZ-3986: --- Need to redo this with a new branch-0.10.0 that is derived from master > Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only) > > > Key: TEZ-3986 > URL: https://issues.apache.org/jira/browse/TEZ-3986 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3986.1.patch > > > Commit only to branch-0.10.0. > Do not commit to master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3986) Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only)
[ https://issues.apache.org/jira/browse/TEZ-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3986: -- Attachment: TEZ-3986.1.patch > Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only) > > > Key: TEZ-3986 > URL: https://issues.apache.org/jira/browse/TEZ-3986 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-3986.1.patch > > > Commit only to branch-0.10.0. > Do not commit to master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3986) Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only)
Eric Wohlstadter created TEZ-3986: - Summary: Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only) Key: TEZ-3986 URL: https://issues.apache.org/jira/browse/TEZ-3986 Project: Apache Tez Issue Type: Task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter Commit only to branch-0.10.0. Do not commit to master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3977) Add Eric Wohlstadter's public key to KEYS
[
https://issues.apache.org/jira/browse/TEZ-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576693#comment-16576693
]
Eric Wohlstadter commented on TEZ-3977:
---
[~jeagles]
Published to [http://pgp.mit.edu/]
Made a backup. Also I backed up a revocation certificate, I can share that to a
secure location if necessary.
> Add Eric Wohlstadter's public key to KEYS
> -
>
> Key: TEZ-3977
> URL: https://issues.apache.org/jira/browse/TEZ-3977
> Project: Apache Tez
> Issue Type: Task
>Reporter: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3977.1.patch
>
>
> {code:java}
> bash> gpg --full-generate-key
> gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Please select what kind of key you want:
> (1) RSA and RSA (default)
> (2) DSA and Elgamal
> (3) DSA (sign only)
> (4) RSA (sign only)
> Your selection? 1
> RSA keys may be between 1024 and 4096 bits long.
> What keysize do you want? (2048) 4096
> Requested keysize is 4096 bits
> Please specify how long the key should be valid.
> 0 = key does not expire
> = key expires in n days
> w = key expires in n weeks
> m = key expires in n months
> y = key expires in n years
> Key is valid for? (0) 0
> Key does not expire at all
> Is this correct? (y/N) y
>
> GnuPG needs to construct a user ID to identify your key.
> Real name: Eric Wohlstadter
> Email address: wohls...@gmail.com
> Comment: CODE SIGNING KEY
> You selected this USER-ID:
> "Eric Wohlstadter (CODE SIGNING KEY) "
> Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
> bash> gpg --send-key X
> gpg: sending key X to hkps://hkps.pool.sks-keyservers.net{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3977) Add Eric Wohlstadter's public key to KEYS
[
https://issues.apache.org/jira/browse/TEZ-3977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-3977:
--
Attachment: TEZ-3977.1.patch
> Add Eric Wohlstadter's public key to KEYS
> -
>
> Key: TEZ-3977
> URL: https://issues.apache.org/jira/browse/TEZ-3977
> Project: Apache Tez
> Issue Type: Task
>Reporter: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3977.1.patch
>
>
> {code:java}
> bash> gpg --full-generate-key
> gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Please select what kind of key you want:
> (1) RSA and RSA (default)
> (2) DSA and Elgamal
> (3) DSA (sign only)
> (4) RSA (sign only)
> Your selection? 1
> RSA keys may be between 1024 and 4096 bits long.
> What keysize do you want? (2048) 4096
> Requested keysize is 4096 bits
> Please specify how long the key should be valid.
> 0 = key does not expire
> = key expires in n days
> w = key expires in n weeks
> m = key expires in n months
> y = key expires in n years
> Key is valid for? (0) 0
> Key does not expire at all
> Is this correct? (y/N) y
>
> GnuPG needs to construct a user ID to identify your key.
> Real name: Eric Wohlstadter
> Email address: wohls...@gmail.com
> Comment: CODE SIGNING KEY
> You selected this USER-ID:
> "Eric Wohlstadter (CODE SIGNING KEY) "
> Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
> bash> gpg --send-key X
> gpg: sending key X to hkps://hkps.pool.sks-keyservers.net{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-3977) Add Eric Wohlstadter's public key to KEYS
Eric Wohlstadter created TEZ-3977:
-
Summary: Add Eric Wohlstadter's public key to KEYS
Key: TEZ-3977
URL: https://issues.apache.org/jira/browse/TEZ-3977
Project: Apache Tez
Issue Type: Task
Reporter: Eric Wohlstadter
{code:java}
bash> gpg --full-generate-key
gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
0 = key does not expire
= key expires in n days
w = key expires in n weeks
m = key expires in n months
y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y
GnuPG needs to construct a user ID to identify your key.
Real name: Eric Wohlstadter
Email address: wohls...@gmail.com
Comment: CODE SIGNING KEY
You selected this USER-ID:
"Eric Wohlstadter (CODE SIGNING KEY) "
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
bash> gpg --send-key X
gpg: sending key X to hkps://hkps.pool.sks-keyservers.net{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3694) Adopt YARN-5007 in MiniTezCluster
[ https://issues.apache.org/jira/browse/TEZ-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548327#comment-16548327 ] Eric Wohlstadter commented on TEZ-3694: --- [~kshukla] Another one of these :) I'm thinking we'll need to drop this from the 0.9.2 and 0.10 release, since the assignee is not active and no one else has picked this up. Do you agree with that? > Adopt YARN-5007 in MiniTezCluster > - > > Key: TEZ-3694 > URL: https://issues.apache.org/jira/browse/TEZ-3694 > Project: Apache Tez > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang >Priority: Major > Attachments: TEZ-3694.1.patch > > > Master branch won't build on hadoop trunk because YARN-5007 removed enableAHS > param from MiniYarnCluster ctor, which breaks MiniTezCluster. We should adopt > the change and use config to enable timeline service. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3718) Better handling of 'bad' nodes
[ https://issues.apache.org/jira/browse/TEZ-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548325#comment-16548325 ] Eric Wohlstadter commented on TEZ-3718: --- [~kshukla] [~jlowe] I'm thinking we'll need to drop this from the 0.9.2 and 0.10 release, since the assignee is not active and no one else has picked this up. Does that sound right to you? > Better handling of 'bad' nodes > -- > > Key: TEZ-3718 > URL: https://issues.apache.org/jira/browse/TEZ-3718 > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Zhiyuan Yang >Priority: Major > Attachments: TEZ-3718.1.patch, TEZ-3718.2.patch, TEZ-3718.3.patch, > TEZ-3718.4.patch > > > At the moment, the default behaviour in case of a node being marked bad is to > do nothing other than not schedule new tasks on this node. > The alternate, via config, is to retroactively kill every task which ran on > the node, which causes far too many unnecessary re-runs. > Proposing the following changes. > 1. KILL fragments which are currently in the RUNNING state (instead of > relying on a timeout which leads to the attempt being marked as FAILED after > the timeout interval. > 2. Keep track of these failed nodes, and use this as input to the failure > heuristics. Normally source tasks require multiple consumers to report > failure for them to be marked as bad. If a single consumer reports failure > against a source which ran on a bad node, consider it bad and re-schedule > immediately. (Otherwise failures can take a while to propagate, and jobs get > a lot slower). > [~jlowe] - think you've looked at this in the past. Any thoughts/suggestions. > What I'm seeing is retroactive failures taking a long time to apply, and > restart sources which ran on a bad node. Also running tasks being counted as > FAILURES instead of KILLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3810) TezCounter for idle time in shuffle phase
[ https://issues.apache.org/jira/browse/TEZ-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3810: -- Target Version/s: 0.9.2, 0.10.0 (was: 0.9.2) > TezCounter for idle time in shuffle phase > - > > Key: TEZ-3810 > URL: https://issues.apache.org/jira/browse/TEZ-3810 > Project: Apache Tez > Issue Type: Improvement >Reporter: Ashwin Ramesh >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-3810-001.patch, TEZ-3810.002.patch, > TEZ-3810.003.patch, TEZ-3810.004.patch, TEZ-3810.005.patch > > > A task attempt counter that tracks how much time was spent waiting for > inputs in the shuffle phase. We can use this to quickly identify jobs that > are wasting a lot of time on the grid with idle reducer tasks instead of > shuffling/merging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3910) Single node can cause Tez job to fail during shuffle
[ https://issues.apache.org/jira/browse/TEZ-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3910: -- Target Version/s: 0.9.2, 0.10.0 (was: 0.9.2) > Single node can cause Tez job to fail during shuffle > > > Key: TEZ-3910 > URL: https://issues.apache.org/jira/browse/TEZ-3910 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-3910.001.patch, TEZ-3910.002.patch, > TEZ-3910.003.patch, TEZ-3910.004.patch, TEZ-3910.005.patch > > > There is a race where a downstream task that is running into fetch failures > due to bad output from the upstream task can continue to blame itself for the > failure before the AM can do a re-run of the upstream offending task and fix > the fetch failure. This causes the DAG to fail even if a single node fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3936) Reduce TezEvent messaging overhead
[ https://issues.apache.org/jira/browse/TEZ-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3936: -- Target Version/s: 0.9.2, 0.10.0 (was: 0.9.2) > Reduce TezEvent messaging overhead > -- > > Key: TEZ-3936 > URL: https://issues.apache.org/jira/browse/TEZ-3936 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Major > Attachments: TEZ-3936.001.patch, TEZ-3936.002.patch > > > Revisiting TEZ-3145, and found that in addition to improving the way empty > partitions are send from Maps to AM and AM to Reducers, message serialization > can be improved to reduce network traffic. > For example in a job with 42000 Maps and 7500 reduces where 95% of the > partition data produced is empty. Tez DME events send from the AM to the > Reducers is num(Maps) * num(Reducers) * size (Wrapped DME). With 95% empty > partitions message size is 450 bytes where 260 bytes is needed for sending > empty partitions and 190 bytes for messaging. Total messaging is 132 GBs > 76 GBs for empty partition data and 56 GBs for non-empty partition messaging. > This jira aims to reduce the non-empty partition messaging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3918) Setting tez.task.log.level does not work
[ https://issues.apache.org/jira/browse/TEZ-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3918: -- Target Version/s: 0.9.2, 0.10.0 (was: 0.9.2) > Setting tez.task.log.level does not work > > > Key: TEZ-3918 > URL: https://issues.apache.org/jira/browse/TEZ-3918 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-3918.001.patch, TEZ-3918.test.patch > > > Setting this as a -D property on cmd line does not take effect. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3916) Add hadoop-azure-datalake jar to azure profile
[ https://issues.apache.org/jira/browse/TEZ-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3916: -- Target Version/s: 0.10.0 Fix Version/s: (was: 0.10.0) > Add hadoop-azure-datalake jar to azure profile > -- > > Key: TEZ-3916 > URL: https://issues.apache.org/jira/browse/TEZ-3916 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Critical > Attachments: TEZ-3916.1.patch, TEZ-3916.2.patch, TEZ-3916.3.patch > > > This jar is required for secure access to Azure object storage: > https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html > There is already an azure profile in Tez but it doesn't include this jar. > Since the jar is only supported on Hadoop 2.8+, will either need to: > 1. Determine that including it in a 2.7 build is fine > 2. Or if it is not fine, then include the jar only when both the 2.8 profile > and the azure profile are activated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3942) RPC getTask writable optimization invalid in hadoop 2.8+
[
https://issues.apache.org/jira/browse/TEZ-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-3942:
--
Target Version/s: 0.9.2, 0.10.0 (was: 0.9.2)
> RPC getTask writable optimization invalid in hadoop 2.8+
>
>
> Key: TEZ-3942
> URL: https://issues.apache.org/jira/browse/TEZ-3942
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-3942.001.patch, TEZ-3942.test.patch
>
>
> TEZ-3140 added an optimization to improve performance of RPC writable.
> HADOOP-13426 added in hadoop 2.8 has invalidated the assumption of the added
> optimization by changing the underlying output buffer.
> {noformat}
> "IPC Server handler 25 on 35274" #85 daemon prio=5 os_prio=0
> tid=0x022c nid=0x1b40f runnable [0x2ba1a6627000]
>java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> - locked <0x00072fe9ac68> (a
> org.apache.hadoop.ipc.ResponseBuffer$FramedBuffer)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> - locked <0x00072fe9ac48> (a org.apache.hadoop.ipc.ResponseBuffer)
> at
> org.apache.tez.dag.api.EntityDescriptor.write(EntityDescriptor.java:121)
> at org.apache.tez.runtime.api.impl.InputSpec.write(InputSpec.java:66)
> at org.apache.tez.runtime.api.impl.TaskSpec.write(TaskSpec.java:174)
> at org.apache.tez.common.ContainerTask.write(ContainerTask.java:77)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:202)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:128)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:82)
> at
> org.apache.hadoop.ipc.RpcWritable$WritableWrapper.writeTo(RpcWritable.java:75)
> at
> org.apache.hadoop.ipc.Server.setupResponseForWritable(Server.java:2807)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:2792)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:2766)
> at org.apache.hadoop.ipc.Server.access$100(Server.java:138)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:905)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:810)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1949)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2523)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3964) Inflater not closed in some places
[
https://issues.apache.org/jira/browse/TEZ-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540591#comment-16540591
]
Eric Wohlstadter commented on TEZ-3964:
---
[~kshukla]
Added Target Version 0.9.2. Please feel free to remove if you don't think it
fits for that release.
> Inflater not closed in some places
> --
>
> Key: TEZ-3964
> URL: https://issues.apache.org/jira/browse/TEZ-3964
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
>
> We call [this
> method|https://github.com/apache/tez/blob/314dfc79b4b3f528b680b4fee73ad0dca3a3a19b/tez-api/src/main/java/org/apache/tez/common/TezCommonUtils.java#L363]
> from a few places. We don't call {{end()}} from most of the place where we
> call and although it's not necessary to call it explicitly it's the
> recommended way in the docs to do so.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3964) Inflater not closed in some places
[
https://issues.apache.org/jira/browse/TEZ-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-3964:
--
Target Version/s: 0.9.2, 0.10.0
> Inflater not closed in some places
> --
>
> Key: TEZ-3964
> URL: https://issues.apache.org/jira/browse/TEZ-3964
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
>
> We call [this
> method|https://github.com/apache/tez/blob/314dfc79b4b3f528b680b4fee73ad0dca3a3a19b/tez-api/src/main/java/org/apache/tez/common/TezCommonUtils.java#L363]
> from a few places. We don't call {{end()}} from most of the place where we
> call and although it's not necessary to call it explicitly it's the
> recommended way in the docs to do so.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3964) Inflater not closed in some places
[
https://issues.apache.org/jira/browse/TEZ-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540588#comment-16540588
]
Eric Wohlstadter commented on TEZ-3964:
---
[~jmarhuen]
When you have a patch addressing review comments, can you attach and submit?
Thanks!
> Inflater not closed in some places
> --
>
> Key: TEZ-3964
> URL: https://issues.apache.org/jira/browse/TEZ-3964
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
>
> We call [this
> method|https://github.com/apache/tez/blob/314dfc79b4b3f528b680b4fee73ad0dca3a3a19b/tez-api/src/main/java/org/apache/tez/common/TezCommonUtils.java#L363]
> from a few places. We don't call {{end()}} from most of the place where we
> call and although it's not necessary to call it explicitly it's the
> recommended way in the docs to do so.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TEZ-3800) Format Hive queries in Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter resolved TEZ-3800. --- Resolution: Won't Fix > Format Hive queries in Tez UI > - > > Key: TEZ-3800 > URL: https://issues.apache.org/jira/browse/TEZ-3800 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Prasanth Jayachandran >Priority: Major > Attachments: Query.png > > > In Hive Queries tab of Tez UI, displayed hive query can be formatted using > sql formatter for better readability of the query string. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-3875) Add "reconnect" API to TezClient for HSI HA
[
https://issues.apache.org/jira/browse/TEZ-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter resolved TEZ-3875.
---
Resolution: Duplicate
Duplicates TEZ-3892
> Add "reconnect" API to TezClient for HSI HA
> ---
>
> Key: TEZ-3875
> URL: https://issues.apache.org/jira/browse/TEZ-3875
> Project: Apache Tez
> Issue Type: New Feature
>Affects Versions: 0.9.next
>Reporter: Eric Wohlstadter
>Priority: Major
>
> For HiveServer2 HA use cases with LLAP, launching a new Tez session (AM)
> after failover adds non-interactive latency. Additionally, Tez with LLAP does
> significant work to provision a long-lived session and hold on to necessary
> resources allocated to it. We don't want to redo that work after failover and
> potentially be unable to schedule equivalent resources that may be taken by
> other applications during failover.
> HS2 uses TezClient for creating sessions. To simplify HS2 failover to an
> existing Tez AM, add a "reconnect" method to TezClient.
> {code}
> public void reconnect(ApplicationId) { ... }
> {code}
> {{reconnect}} will setup the TezClient context very similar to {{start}}
> except it parameterizes the {{sessionAppId}} and doesn't launch a new AM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3800) Format Hive queries in Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540561#comment-16540561 ] Eric Wohlstadter commented on TEZ-3800: --- [~prasanth_j] Can we close this as won't fix? > Format Hive queries in Tez UI > - > > Key: TEZ-3800 > URL: https://issues.apache.org/jira/browse/TEZ-3800 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Prasanth Jayachandran >Priority: Major > Attachments: Query.png > > > In Hive Queries tab of Tez UI, displayed hive query can be formatted using > sql formatter for better readability of the query string. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3942) RPC getTask writable optimization invalid in hadoop 2.8+
[
https://issues.apache.org/jira/browse/TEZ-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540549#comment-16540549
]
Eric Wohlstadter commented on TEZ-3942:
---
[~jeagles]
Do you know if this is applicable for Hadoop3, e.g. 0.10 release?
> RPC getTask writable optimization invalid in hadoop 2.8+
>
>
> Key: TEZ-3942
> URL: https://issues.apache.org/jira/browse/TEZ-3942
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-3942.001.patch, TEZ-3942.test.patch
>
>
> TEZ-3140 added an optimization to improve performance of RPC writable.
> HADOOP-13426 added in hadoop 2.8 has invalidated the assumption of the added
> optimization by changing the underlying output buffer.
> {noformat}
> "IPC Server handler 25 on 35274" #85 daemon prio=5 os_prio=0
> tid=0x022c nid=0x1b40f runnable [0x2ba1a6627000]
>java.lang.Thread.State: RUNNABLE
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> - locked <0x00072fe9ac68> (a
> org.apache.hadoop.ipc.ResponseBuffer$FramedBuffer)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> - locked <0x00072fe9ac48> (a org.apache.hadoop.ipc.ResponseBuffer)
> at
> org.apache.tez.dag.api.EntityDescriptor.write(EntityDescriptor.java:121)
> at org.apache.tez.runtime.api.impl.InputSpec.write(InputSpec.java:66)
> at org.apache.tez.runtime.api.impl.TaskSpec.write(TaskSpec.java:174)
> at org.apache.tez.common.ContainerTask.write(ContainerTask.java:77)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:202)
> at
> org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:128)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:82)
> at
> org.apache.hadoop.ipc.RpcWritable$WritableWrapper.writeTo(RpcWritable.java:75)
> at
> org.apache.hadoop.ipc.Server.setupResponseForWritable(Server.java:2807)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:2792)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:2766)
> at org.apache.hadoop.ipc.Server.access$100(Server.java:138)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:905)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:810)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1949)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2523)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3916) Add hadoop-azure-datalake jar to azure profile
[ https://issues.apache.org/jira/browse/TEZ-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3916: -- Fix Version/s: 0.10.0 > Add hadoop-azure-datalake jar to azure profile > -- > > Key: TEZ-3916 > URL: https://issues.apache.org/jira/browse/TEZ-3916 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Critical > Fix For: 0.10.0 > > Attachments: TEZ-3916.1.patch, TEZ-3916.2.patch, TEZ-3916.3.patch > > > This jar is required for secure access to Azure object storage: > https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html > There is already an azure profile in Tez but it doesn't include this jar. > Since the jar is only supported on Hadoop 2.8+, will either need to: > 1. Determine that including it in a 2.7 build is fine > 2. Or if it is not fine, then include the jar only when both the 2.8 profile > and the azure profile are activated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3916) Add hadoop-azure-datalake jar to azure profile
[ https://issues.apache.org/jira/browse/TEZ-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3916: -- Fix Version/s: (was: 0.10.0) > Add hadoop-azure-datalake jar to azure profile > -- > > Key: TEZ-3916 > URL: https://issues.apache.org/jira/browse/TEZ-3916 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Critical > Fix For: 0.10.0 > > Attachments: TEZ-3916.1.patch, TEZ-3916.2.patch, TEZ-3916.3.patch > > > This jar is required for secure access to Azure object storage: > https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html > There is already an azure profile in Tez but it doesn't include this jar. > Since the jar is only supported on Hadoop 2.8+, will either need to: > 1. Determine that including it in a 2.7 build is fine > 2. Or if it is not fine, then include the jar only when both the 2.8 profile > and the azure profile are activated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-3903) [Umbrella] Hadoop 3 dependency changes
[ https://issues.apache.org/jira/browse/TEZ-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter resolved TEZ-3903. --- Resolution: Fixed Target Version/s: 0.10.0 (was: 0.9.2) Resolving as all child tickets are resolved. > [Umbrella] Hadoop 3 dependency changes > -- > > Key: TEZ-3903 > URL: https://issues.apache.org/jira/browse/TEZ-3903 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > > Placeholder to organize proposed dependency change tickets for 0.10 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3916) Add hadoop-azure-datalake jar to azure profile
[ https://issues.apache.org/jira/browse/TEZ-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540447#comment-16540447 ] Eric Wohlstadter commented on TEZ-3916: --- Resubmitting patch. Expecting no test failures this time. > Add hadoop-azure-datalake jar to azure profile > -- > > Key: TEZ-3916 > URL: https://issues.apache.org/jira/browse/TEZ-3916 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Critical > Fix For: 0.10.0 > > Attachments: TEZ-3916.1.patch, TEZ-3916.2.patch, TEZ-3916.3.patch > > > This jar is required for secure access to Azure object storage: > https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html > There is already an azure profile in Tez but it doesn't include this jar. > Since the jar is only supported on Hadoop 2.8+, will either need to: > 1. Determine that including it in a 2.7 build is fine > 2. Or if it is not fine, then include the jar only when both the 2.8 profile > and the azure profile are activated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3916) Add hadoop-azure-datalake jar to azure profile
[ https://issues.apache.org/jira/browse/TEZ-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3916: -- Attachment: TEZ-3916.3.patch > Add hadoop-azure-datalake jar to azure profile > -- > > Key: TEZ-3916 > URL: https://issues.apache.org/jira/browse/TEZ-3916 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Critical > Fix For: 0.10.0 > > Attachments: TEZ-3916.1.patch, TEZ-3916.2.patch, TEZ-3916.3.patch > > > This jar is required for secure access to Azure object storage: > https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html > There is already an azure profile in Tez but it doesn't include this jar. > Since the jar is only supported on Hadoop 2.8+, will either need to: > 1. Determine that including it in a 2.7 build is fine > 2. Or if it is not fine, then include the jar only when both the 2.8 profile > and the azure profile are activated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-3884) Hadoop3-beta1 fixes for Tez tests
[
https://issues.apache.org/jira/browse/TEZ-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter resolved TEZ-3884.
---
Resolution: Duplicate
> Hadoop3-beta1 fixes for Tez tests
> -
>
> Key: TEZ-3884
> URL: https://issues.apache.org/jira/browse/TEZ-3884
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Gopal V
>Priority: Minor
> Attachments: TEZ-3884.patch.1
>
>
> {code}
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[48,30]
> cannot find symbol
> [ERROR] symbol: class DistributedFileSystem
> [ERROR] location: package org.apache.hadoop.hdfs
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[680,50]
> cannot find symbol
> [ERROR] symbol: class DistributedFileSystem
> [ERROR] location: class org.apache.tez.client.TestTezClientUtils
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/common/TestTezCommonUtils.java:[62,42]
> cannot access org.apache.hadoop.hdfs.DistributedFileSystem
> [ERROR] class file for org.apache.hadoop.hdfs.DistributedFileSystem not found
> [ERROR] -> [Help 1]
> [ERROR]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3955) Upgrade hadoop dependency to 3.0.3
[ https://issues.apache.org/jira/browse/TEZ-3955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16539101#comment-16539101 ] Eric Wohlstadter commented on TEZ-3955: --- +1 (unbinding) > Upgrade hadoop dependency to 3.0.3 > -- > > Key: TEZ-3955 > URL: https://issues.apache.org/jira/browse/TEZ-3955 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Major > Attachments: TEZ-3955.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3916) Add hadoop-azure-datalake jar to azure profile
[ https://issues.apache.org/jira/browse/TEZ-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537724#comment-16537724 ] Eric Wohlstadter commented on TEZ-3916: --- [~gopalv] The test failure is expected because of HADOOP-15450, which will be fixed by TEZ-3955. > Add hadoop-azure-datalake jar to azure profile > -- > > Key: TEZ-3916 > URL: https://issues.apache.org/jira/browse/TEZ-3916 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Critical > Fix For: 0.10.0 > > Attachments: TEZ-3916.1.patch, TEZ-3916.2.patch > > > This jar is required for secure access to Azure object storage: > https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html > There is already an azure profile in Tez but it doesn't include this jar. > Since the jar is only supported on Hadoop 2.8+, will either need to: > 1. Determine that including it in a 2.7 build is fine > 2. Or if it is not fine, then include the jar only when both the 2.8 profile > and the azure profile are activated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3916) Add hadoop-azure-datalake jar to azure profile
[ https://issues.apache.org/jira/browse/TEZ-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3916: -- Attachment: TEZ-3916.2.patch > Add hadoop-azure-datalake jar to azure profile > -- > > Key: TEZ-3916 > URL: https://issues.apache.org/jira/browse/TEZ-3916 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Critical > Fix For: 0.10.0 > > Attachments: TEZ-3916.1.patch, TEZ-3916.2.patch > > > This jar is required for secure access to Azure object storage: > https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html > There is already an azure profile in Tez but it doesn't include this jar. > Since the jar is only supported on Hadoop 2.8+, will either need to: > 1. Determine that including it in a 2.7 build is fine > 2. Or if it is not fine, then include the jar only when both the 2.8 profile > and the azure profile are activated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3958) Add internal vertex priority information into the tez dag.dot debug information
[
https://issues.apache.org/jira/browse/TEZ-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535461#comment-16535461
]
Eric Wohlstadter commented on TEZ-3958:
---
[~jmarhuen]
We should avoid introducing a new {{abstract}} method into {{DAGScheduler}}.
This class might be extended by outside applications which are using Tez, and
we don't want to break them. I would give {{getPriorityLowLimit}} a default
implementation. Something like {{throw new UnsupportedOperationException, and}}
then skip generation of priorities in the .dot file if this exception is thrown.
Also this logic:
{code:java}
final int vertexDistanceFromRoot = vertex.getDistanceFromRoot();
return ((vertexDistanceFromRoot + 1) * dag.getTotalVertices() * 3)
+ (vertex.getVertexId().getId() * 3);{code}
is repeated in two places in the patch, and also in both
{{DAGSchedulerNaturalOrder}} and {{DAGSchedulerNaturalOrderControlled}}. This
looks like it should be refactored into the base class, so that it appears one
time instead of four. Again we need to be careful here about not breaking
sub-classes of {{DAGScheduler}} which are not in the Tez project source code.
> Add internal vertex priority information into the tez dag.dot debug
> information
> ---
>
> Key: TEZ-3958
> URL: https://issues.apache.org/jira/browse/TEZ-3958
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Gopal V
>Assignee: Jaume M
>Priority: Major
> Attachments: TEZ-3958.1.patch, TEZ-3958.2.patch
>
>
> Adding the actual vertex priority as computed by Tez into the debug dag.dot
> file would allows the debugging of task pre-emption issues when the DAG is no
> longer a tree.
> There are pre-emption issues with isomerization of Tez DAGs, where the a
> R-isomer dag with mirror rotation runs at a different speed than the L-isomer
> dag, due to priorities at the same level changing due to the vertex-id order.
> Since the problem is hard to debug through, it would be good to record the
> computed priority in the DAG .dot file in the logging directories.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3962) Configuration decode leaks an Inflater object
[
https://issues.apache.org/jira/browse/TEZ-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526673#comment-16526673
]
Eric Wohlstadter commented on TEZ-3962:
---
[~jlowe]
Can you take a look at patch 2? Thanks for your help.
> Configuration decode leaks an Inflater object
> -
>
> Key: TEZ-3962
> URL: https://issues.apache.org/jira/browse/TEZ-3962
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.2, 0.10.0
>Reporter: Gopal V
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3962.1.patch, TEZ-3962.2.patch
>
>
> {code}
> public static Configuration createConfFromByteString(ByteString byteString)
> throws IOException {
> ...
> InflaterInputStream uncompressIs = new
> InflaterInputStream(byteString.newInput());
> DAGProtos.ConfigurationProto confProto =
> DAGProtos.ConfigurationProto.parseFrom(uncompressIs);
> {code}
> InflaterInputStream is never closed, this will get eventually collected - but
> the off-heap buffers for Inflater leaks temporarily.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3962) Configuration decode leaks an Inflater object
[
https://issues.apache.org/jira/browse/TEZ-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-3962:
--
Attachment: TEZ-3962.2.patch
> Configuration decode leaks an Inflater object
> -
>
> Key: TEZ-3962
> URL: https://issues.apache.org/jira/browse/TEZ-3962
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.2, 0.10.0
>Reporter: Gopal V
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3962.1.patch, TEZ-3962.2.patch
>
>
> {code}
> public static Configuration createConfFromByteString(ByteString byteString)
> throws IOException {
> ...
> InflaterInputStream uncompressIs = new
> InflaterInputStream(byteString.newInput());
> DAGProtos.ConfigurationProto confProto =
> DAGProtos.ConfigurationProto.parseFrom(uncompressIs);
> {code}
> InflaterInputStream is never closed, this will get eventually collected - but
> the off-heap buffers for Inflater leaks temporarily.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3962) Configuration decode leaks an Inflater object
[
https://issues.apache.org/jira/browse/TEZ-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526661#comment-16526661
]
Eric Wohlstadter commented on TEZ-3962:
---
[~jlowe]
Thanks for the review. I see what you mean. The new code introduces a new
failure mode since now {{close}} may fail. But we can just let an exception
from {{close}} be propagated without any specific intervention.
> Configuration decode leaks an Inflater object
> -
>
> Key: TEZ-3962
> URL: https://issues.apache.org/jira/browse/TEZ-3962
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.2, 0.10.0
>Reporter: Gopal V
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3962.1.patch
>
>
> {code}
> public static Configuration createConfFromByteString(ByteString byteString)
> throws IOException {
> ...
> InflaterInputStream uncompressIs = new
> InflaterInputStream(byteString.newInput());
> DAGProtos.ConfigurationProto confProto =
> DAGProtos.ConfigurationProto.parseFrom(uncompressIs);
> {code}
> InflaterInputStream is never closed, this will get eventually collected - but
> the off-heap buffers for Inflater leaks temporarily.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (TEZ-3962) Configuration decode leaks an Inflater object
[
https://issues.apache.org/jira/browse/TEZ-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter reassigned TEZ-3962:
-
Assignee: Eric Wohlstadter
> Configuration decode leaks an Inflater object
> -
>
> Key: TEZ-3962
> URL: https://issues.apache.org/jira/browse/TEZ-3962
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.2, 0.10.0
>Reporter: Gopal V
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: TEZ-3962.1.patch
>
>
> {code}
> public static Configuration createConfFromByteString(ByteString byteString)
> throws IOException {
> ...
> InflaterInputStream uncompressIs = new
> InflaterInputStream(byteString.newInput());
> DAGProtos.ConfigurationProto confProto =
> DAGProtos.ConfigurationProto.parseFrom(uncompressIs);
> {code}
> InflaterInputStream is never closed, this will get eventually collected - but
> the off-heap buffers for Inflater leaks temporarily.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3962) Configuration decode leaks an Inflater object
[
https://issues.apache.org/jira/browse/TEZ-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated TEZ-3962:
--
Attachment: TEZ-3962.1.patch
> Configuration decode leaks an Inflater object
> -
>
> Key: TEZ-3962
> URL: https://issues.apache.org/jira/browse/TEZ-3962
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.2, 0.10.0
>Reporter: Gopal V
>Priority: Major
> Attachments: TEZ-3962.1.patch
>
>
> {code}
> public static Configuration createConfFromByteString(ByteString byteString)
> throws IOException {
> ...
> InflaterInputStream uncompressIs = new
> InflaterInputStream(byteString.newInput());
> DAGProtos.ConfigurationProto confProto =
> DAGProtos.ConfigurationProto.parseFrom(uncompressIs);
> {code}
> InflaterInputStream is never closed, this will get eventually collected - but
> the off-heap buffers for Inflater leaks temporarily.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3959) HTTP 502 for bower install
[
https://issues.apache.org/jira/browse/TEZ-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522493#comment-16522493
]
Eric Wohlstadter commented on TEZ-3959:
---
/cc [~jeagles]
> HTTP 502 for bower install
> --
>
> Key: TEZ-3959
> URL: https://issues.apache.org/jira/browse/TEZ-3959
> Project: Apache Tez
> Issue Type: Bug
> Components: UI
>Reporter: Eric Wohlstadter
>Assignee: Harish Jaiprakash
>Priority: Blocker
> Attachments: TEZ-3959.01.patch, TEZ-3959.02.patch
>
>
> [~harishjp] and I started seeing:
> {code:java}
> 18:56:57 2018/06/24 01:56:57 INFO: [ERROR] bower moment#2.12.0
> EINVRES Request to https://bower.herokuapp.com/packages/moment failed with
> 502{code}
> [~harishjp] did some digging and found that changing to version 1.8.4 seems
> to be a workaround, but it is dubious:
> {code}
> ERROR warning bower@1.8.4: We don't recommend using Bower for new projects.
> Please consider Yarn and Webpack or Parcel. You can read how to migrate
> legacy project here:
> https://bower.io/blog/2017/how-to-migrate-away-from-bower/
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-3959) HTTP 502 for bower install
Eric Wohlstadter created TEZ-3959:
-
Summary: HTTP 502 for bower install
Key: TEZ-3959
URL: https://issues.apache.org/jira/browse/TEZ-3959
Project: Apache Tez
Issue Type: Bug
Components: UI
Reporter: Eric Wohlstadter
[~harishjp] and I started seeing:
{code:java}
18:56:57 2018/06/24 01:56:57 INFO: [ERROR] bower moment#2.12.0
EINVRES Request to https://bower.herokuapp.com/packages/moment failed with
502{code}
[~harishjp] did some digging and found that changing to version 1.8.4 seems to
be a workaround, but it is dubious:
{code}
ERROR warning bower@1.8.4: We don't recommend using Bower for new projects.
Please consider Yarn and Webpack or Parcel. You can read how to migrate legacy
project here: https://bower.io/blog/2017/how-to-migrate-away-from-bower/
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
[
https://issues.apache.org/jira/browse/TEZ-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517604#comment-16517604
]
Eric Wohlstadter commented on TEZ-3957:
---
I realize we can't just keep adding Counters for everything, but this one seems
like it would be a very common use-case.
> Report TASK_DURATION_MILLIS as a Counter for completed tasks
>
>
> Key: TEZ-3957
> URL: https://issues.apache.org/jira/browse/TEZ-3957
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
>
> timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not
> as a Counter.
> Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
> avg(timeTaken).
> The value will be: {{finishTime - launchTime}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-3957) Report TASK_DURATION_MILLIS as a Counter for completed tasks
Eric Wohlstadter created TEZ-3957:
-
Summary: Report TASK_DURATION_MILLIS as a Counter for completed
tasks
Key: TEZ-3957
URL: https://issues.apache.org/jira/browse/TEZ-3957
Project: Apache Tez
Issue Type: Improvement
Reporter: Eric Wohlstadter
Assignee: Eric Wohlstadter
timeTaken is already being reported by {{TaskAttemptFinishedEvent}}, but not as
a Counter.
Combined with TEZ-3911, this provides min(timeTaken), max(timeTaken),
avg(timeTaken).
The value will be: {{finishTime - launchTime}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3923) Move master to Hadoop 3+ and create separate 0.9.x line
[ https://issues.apache.org/jira/browse/TEZ-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506273#comment-16506273 ] Eric Wohlstadter commented on TEZ-3923: --- [~jeagles] Are there any remaining blockers for cutting a release from master? Can we make a call on the mailing list now? > Move master to Hadoop 3+ and create separate 0.9.x line > --- > > Key: TEZ-3923 > URL: https://issues.apache.org/jira/browse/TEZ-3923 > Project: Apache Tez > Issue Type: Task >Reporter: Eric Wohlstadter >Assignee: Jonathan Eagles >Priority: Major > Fix For: 0.10.0 > > Attachments: TEZ-3923.001.patch, TEZ-3923.002.patch > > > Move master to support minimum Hadoop 3+ (0.10.x line) and create separate > branch for Hadoop 2 (0.9.x line) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3951) TezClient wait too long for the DAGClient for prewarm; tries to shut down the wrong DAG
[ https://issues.apache.org/jira/browse/TEZ-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506227#comment-16506227 ] Eric Wohlstadter commented on TEZ-3951: --- lgtm > TezClient wait too long for the DAGClient for prewarm; tries to shut down the > wrong DAG > --- > > Key: TEZ-3951 > URL: https://issues.apache.org/jira/browse/TEZ-3951 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: TEZ-3951.01.patch, TEZ-3951.patch > > > Follow-up from TEZ-3943 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
