[jira] [Created] (TEZ-3875) Add "reconnect" API to TezClient for HSI HA
Eric Wohlstadter created TEZ-3875:
-
Summary: Add "reconnect" API to TezClient for HSI HA
Key: TEZ-3875
URL: https://issues.apache.org/jira/browse/TEZ-3875
Project: Apache Tez
Issue Type: New Feature
Affects Versions: 0.9.next
Reporter: Eric Wohlstadter
For HiveServer2 HA use cases with LLAP, launching a new Tez session (AM) after
failover adds non-interactive latency. Additionally, Tez with LLAP does
significant work to provision a long-lived session and hold on to necessary
resources allocated to it. We don't want to redo that work after failover and
potentially be unable to schedule equivalent resources that may be taken by
other applications during failover.
HS2 uses TezClient for creating sessions. To simplify HS2 failover to an
existing Tez AM, add a "reconnect" method to TezClient.
{code}
public void reconnect(ApplicationId) { ... }
{code}
{{reconnect}} will setup the TezClient context very similar to {{start}} except
it parameterizes the {{sessionAppId}} and doesn't launch a new AM.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (TEZ-3874) NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in Configuration
Eric Wohlstadter created TEZ-3874:
-
Summary: NPE in TezClientUtils when
"yarn.resourcemanager.zk-address" is present in Configuration
Key: TEZ-3874
URL: https://issues.apache.org/jira/browse/TEZ-3874
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.9.1
Reporter: Eric Wohlstadter
Priority: Blocker
"yarn.resourcemanager.zk-address" is deprecated in favor of "hadoop.zk.address"
for Hadoop 2.9+.
Configuration base class does't auto-translate the deprecation. Only
YarnConfiguration applies the translation.
In TezClientUtils.createFinalConfProtoForApp, a NPE is throw if
"yarn.resourcemanager.zk-address" is present in the Configuration.
{code}
for (Entry entry : amConf) {
PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
kvp.setKey(entry.getKey());
kvp.setValue(amConf.get(entry.getKey()));
builder.addConfKeyValues(kvp);
}
{code}
Even though Tez is not specifically looking for the deprecated property,
{{amConf.get(entry.getKey())}} will find it during the iteration, if it is in
any of the merged xml property resources.
{{amConf.get(entry.getKey())}} will return null, and {{kvp.setValue(null)}}
will trigger NPE.
Suggested solution is to change to:
{code}
YarnConfiguration wrappedConf = new YarnConfiguration(amConf);
for (Entry entry : wrappedConf) {
PlanKeyValuePair.Builder kvp = PlanKeyValuePair.newBuilder();
kvp.setKey(entry.getKey());
kvp.setValue(wrappedConf.get(entry.getKey()));
builder.addConfKeyValues(kvp);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (TEZ-3944) TestTaskScheduler
Eric Wohlstadter created TEZ-3944: - Summary: TestTaskScheduler Key: TEZ-3944 URL: https://issues.apache.org/jira/browse/TEZ-3944 Project: Apache Tez Issue Type: Bug Reporter: Eric Wohlstadter TestTaskScheduler times-out intermittently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-3945) TestHistoryParser fails with Hadoop3 build
[
https://issues.apache.org/jira/browse/TEZ-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter resolved TEZ-3945.
---
Resolution: Duplicate
Resolving as Duplicate of TEZ-3946
> TestHistoryParser fails with Hadoop3 build
> --
>
> Key: TEZ-3945
> URL: https://issues.apache.org/jira/browse/TEZ-3945
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Priority: Major
>
> Seems to be related to connection with AHS.
> {code}
> 2018-05-29 19:47:18,281 INFO [main] client.AHSProxy
> (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server
> at /0.0.0.0:10200
> 2018-05-29 19:47:19,314 INFO [main] ipc.Client
> (Client.java:handleConnectionFailure(940)) - Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> {code}
> then I see retrying occur continuously.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-3959) HTTP 502 for bower install
Eric Wohlstadter created TEZ-3959:
-
Summary: HTTP 502 for bower install
Key: TEZ-3959
URL: https://issues.apache.org/jira/browse/TEZ-3959
Project: Apache Tez
Issue Type: Bug
Components: UI
Reporter: Eric Wohlstadter
[~harishjp] and I started seeing:
{code:java}
18:56:57 2018/06/24 01:56:57 INFO: [ERROR] bower moment#2.12.0
EINVRES Request to https://bower.herokuapp.com/packages/moment failed with
502{code}
[~harishjp] did some digging and found that changing to version 1.8.4 seems to
be a workaround, but it is dubious:
{code}
ERROR warning bower@1.8.4: We don't recommend using Bower for new projects.
Please consider Yarn and Webpack or Parcel. You can read how to migrate legacy
project here: https://bower.io/blog/2017/how-to-migrate-away-from-bower/
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-3929) Upgrade Jersey to 1.19
Eric Wohlstadter created TEZ-3929: - Summary: Upgrade Jersey to 1.19 Key: TEZ-3929 URL: https://issues.apache.org/jira/browse/TEZ-3929 Project: Apache Tez Issue Type: Task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter Align Jersey version with Hadoop3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3889) flaky test: TestHttpConnection.testAsyncHttpConnectionInterrupt
Eric Wohlstadter created TEZ-3889:
-
Summary: flaky test:
TestHttpConnection.testAsyncHttpConnectionInterrupt
Key: TEZ-3889
URL: https://issues.apache.org/jira/browse/TEZ-3889
Project: Apache Tez
Issue Type: Bug
Affects Versions: 0.9.1
Reporter: Eric Wohlstadter
This test fails for me sometimes and passes sometimes.
Using Hadoop3 with 0.9.2-SNAPSHOT
{code:java}
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.648 sec <<<
FAILURE!
testAsyncHttpConnectionInterrupt(org.apache.tez.http.TestHttpConnection) Time
elapsed: 1.345 sec <<< ERROR!
java.util.concurrent.ExecutionException: java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.tez.http.TestHttpConnection$Worker.call(TestHttpConnection.java:203)
at
org.apache.tez.http.TestHttpConnection$Worker.call(TestHttpConnection.java:177)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Results :
Tests in error:
TestHttpConnection.testAsyncHttpConnectionInterrupt:140 » Execution
java.lang{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-3890) Jenkins test patch scripts don't post results to Tez JIRA
Eric Wohlstadter created TEZ-3890:
-
Summary: Jenkins test patch scripts don't post results to Tez JIRA
Key: TEZ-3890
URL: https://issues.apache.org/jira/browse/TEZ-3890
Project: Apache Tez
Issue Type: Bug
Reporter: Eric Wohlstadter
Jenkins build no longer posts test results to JIRA. The test hook runs and the
results are included in the Jenkins log, but not on JIRA.
{code:java}
==
==
Adding comment to Jira.
==
==
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404
Unable to log in to server:
https://issues.apache.org/jira/rpc/soap/jirasoapservice-v2 with user: tezqa.
Cause: (404)404{code}
>From what I gather from [~aw], some older APIs that were being used by the Tez
>build process have been removed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-3888) Update Jetty to org.eclipse.jetty 9.x
Eric Wohlstadter created TEZ-3888: - Summary: Update Jetty to org.eclipse.jetty 9.x Key: TEZ-3888 URL: https://issues.apache.org/jira/browse/TEZ-3888 Project: Apache Tez Issue Type: Improvement Reporter: Eric Wohlstadter mortbay Jetty 6 is no longer supported and has multiple CVEs. Tez can't be used in scenarios where compliance against vulnerability scanning tools is required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3892) Tez-AM Daemon-Mode
Eric Wohlstadter created TEZ-3892:
-
Summary: Tez-AM Daemon-Mode
Key: TEZ-3892
URL: https://issues.apache.org/jira/browse/TEZ-3892
Project: Apache Tez
Issue Type: New Feature
Reporter: Eric Wohlstadter
This is a proposed opt-in feature.
Tez AM already supports long-lived sessions, if desired a AM session can live
indefinitely.
However, new clients cannot connect to a long-lived AM session through the
standard TezClient API.
TezClient API only provides a "start" method to initiate a connection, which
always allocates a new AM from YARN.
# For interactive BI use-cases, this startup time can be significant.
# Hive is implementing a HiveServer2 High Availability feature.
** When the singleton HS2 master server fails, the HS2 client is quickly
redirected to a pre-warmed HS2 backup.
# For the failover to complete quickly end-to-end, a Tez AM must also be
pre-warmed and ready to accept connections.
For more information, see design for:
https://issues.apache.org/jira/browse/HIVE-18281.
Anticipated changes:
# A {{reconnect(ApplicationId)}} method is added to TezClient. The
functionality is similar to {{start}}
** Code related to launching a new AM from the RM is factored out.
** Since {{start}} and {{reconnect}} will share some code, this code is
refactored into reusable helper methods.
** A usage example is added to {{org/apache/tez/examples}}
# It is not a goal of this JIRA to ensure that running Tez DAGs can be
recovered by a client using the {{reconnect}} API. The goal is only for
maintaining a pool of warm Tez AMs to skip RM/container/JVM startup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-3891) Migrate patch submisssion scripts and hooks to Yetus 0.7.0
Eric Wohlstadter created TEZ-3891: - Summary: Migrate patch submisssion scripts and hooks to Yetus 0.7.0 Key: TEZ-3891 URL: https://issues.apache.org/jira/browse/TEZ-3891 Project: Apache Tez Issue Type: Improvement Reporter: Eric Wohlstadter Patch test/validation results are no longer posted to JIRA. This is due to EOL for some APIs that were used being used. Discussed with [~jlowe] and [~jeagles]. As suggested by [~aw], moving to Yetus 0.7.0 seems to the most sense, rather than try to workaround and carry forward the older scripts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3977) Add Eric Wohlstadter's public key to KEYS
Eric Wohlstadter created TEZ-3977:
-
Summary: Add Eric Wohlstadter's public key to KEYS
Key: TEZ-3977
URL: https://issues.apache.org/jira/browse/TEZ-3977
Project: Apache Tez
Issue Type: Task
Reporter: Eric Wohlstadter
{code:java}
bash> gpg --full-generate-key
gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
0 = key does not expire
= key expires in n days
w = key expires in n weeks
m = key expires in n months
y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y
GnuPG needs to construct a user ID to identify your key.
Real name: Eric Wohlstadter
Email address: wohls...@gmail.com
Comment: CODE SIGNING KEY
You selected this USER-ID:
"Eric Wohlstadter (CODE SIGNING KEY) "
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
bash> gpg --send-key X
gpg: sending key X to hkps://hkps.pool.sks-keyservers.net{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-3986) Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only)
Eric Wohlstadter created TEZ-3986: - Summary: Change branch-0.10.0 artifact versions, add CHANGES.txt (branch-0.10.0 only) Key: TEZ-3986 URL: https://issues.apache.org/jira/browse/TEZ-3986 Project: Apache Tez Issue Type: Task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter Commit only to branch-0.10.0. Do not commit to master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3988) Update snapshot version in master to 0.10.1-SNAPSHOT
Eric Wohlstadter created TEZ-3988: - Summary: Update snapshot version in master to 0.10.1-SNAPSHOT Key: TEZ-3988 URL: https://issues.apache.org/jira/browse/TEZ-3988 Project: Apache Tez Issue Type: Task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-3884) Hadoop3-beta1 fixes for Tez tests
[
https://issues.apache.org/jira/browse/TEZ-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter resolved TEZ-3884.
---
Resolution: Duplicate
> Hadoop3-beta1 fixes for Tez tests
> -
>
> Key: TEZ-3884
> URL: https://issues.apache.org/jira/browse/TEZ-3884
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Gopal V
>Priority: Minor
> Attachments: TEZ-3884.patch.1
>
>
> {code}
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[48,30]
> cannot find symbol
> [ERROR] symbol: class DistributedFileSystem
> [ERROR] location: package org.apache.hadoop.hdfs
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[680,50]
> cannot find symbol
> [ERROR] symbol: class DistributedFileSystem
> [ERROR] location: class org.apache.tez.client.TestTezClientUtils
> [ERROR]
> /grid/5/dev/gopalv/llap-autobuild/tez/tez-api/src/test/java/org/apache/tez/common/TestTezCommonUtils.java:[62,42]
> cannot access org.apache.hadoop.hdfs.DistributedFileSystem
> [ERROR] class file for org.apache.hadoop.hdfs.DistributedFileSystem not found
> [ERROR] -> [Help 1]
> [ERROR]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TEZ-3903) [Umbrella] Hadoop 3 dependency changes
[ https://issues.apache.org/jira/browse/TEZ-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter resolved TEZ-3903. --- Resolution: Fixed Target Version/s: 0.10.0 (was: 0.9.2) Resolving as all child tickets are resolved. > [Umbrella] Hadoop 3 dependency changes > -- > > Key: TEZ-3903 > URL: https://issues.apache.org/jira/browse/TEZ-3903 > Project: Apache Tez > Issue Type: Improvement >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > > Placeholder to organize proposed dependency change tickets for 0.10 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-3875) Add "reconnect" API to TezClient for HSI HA
[
https://issues.apache.org/jira/browse/TEZ-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter resolved TEZ-3875.
---
Resolution: Duplicate
Duplicates TEZ-3892
> Add "reconnect" API to TezClient for HSI HA
> ---
>
> Key: TEZ-3875
> URL: https://issues.apache.org/jira/browse/TEZ-3875
> Project: Apache Tez
> Issue Type: New Feature
>Affects Versions: 0.9.next
>Reporter: Eric Wohlstadter
>Priority: Major
>
> For HiveServer2 HA use cases with LLAP, launching a new Tez session (AM)
> after failover adds non-interactive latency. Additionally, Tez with LLAP does
> significant work to provision a long-lived session and hold on to necessary
> resources allocated to it. We don't want to redo that work after failover and
> potentially be unable to schedule equivalent resources that may be taken by
> other applications during failover.
> HS2 uses TezClient for creating sessions. To simplify HS2 failover to an
> existing Tez AM, add a "reconnect" method to TezClient.
> {code}
> public void reconnect(ApplicationId) { ... }
> {code}
> {{reconnect}} will setup the TezClient context very similar to {{start}}
> except it parameterizes the {{sessionAppId}} and doesn't launch a new AM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TEZ-3800) Format Hive queries in Tez UI
[ https://issues.apache.org/jira/browse/TEZ-3800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter resolved TEZ-3800. --- Resolution: Won't Fix > Format Hive queries in Tez UI > - > > Key: TEZ-3800 > URL: https://issues.apache.org/jira/browse/TEZ-3800 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Prasanth Jayachandran >Priority: Major > Attachments: Query.png > > > In Hive Queries tab of Tez UI, displayed hive query can be formatted using > sql formatter for better readability of the query string. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3902) Upgrade to netty-3.10.5.Final.jar
Eric Wohlstadter created TEZ-3902: - Summary: Upgrade to netty-3.10.5.Final.jar Key: TEZ-3902 URL: https://issues.apache.org/jira/browse/TEZ-3902 Project: Apache Tez Issue Type: Improvement Reporter: Eric Wohlstadter Hadoop 3 and Hive have upgraded to netty-3.10.5.Final, which is not compatible with current Tez dependency netty-3.6.2.Final. However, org.apache.tez.shufflehandler.ShuffleHandler depends on 3.6.2 specific methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3903) [Umbrella] 0.9.2 maven deps. changes
Eric Wohlstadter created TEZ-3903: - Summary: [Umbrella] 0.9.2 maven deps. changes Key: TEZ-3903 URL: https://issues.apache.org/jira/browse/TEZ-3903 Project: Apache Tez Issue Type: Improvement Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter Placeholder to organize proposed dependency change tickets for 0.9.2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3901) Add hadoop3 profile for upgrade to Jersey 1.19
Eric Wohlstadter created TEZ-3901: - Summary: Add hadoop3 profile for upgrade to Jersey 1.19 Key: TEZ-3901 URL: https://issues.apache.org/jira/browse/TEZ-3901 Project: Apache Tez Issue Type: Improvement Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter >From [~harishjp]: "DAGAppMaster fails to start when using hadoop3 and ATSv15, because TimelineWriter has been changed to use jersey-client 1.19 in hadoop3, but tez packages jersey-client 1.9 with it. There are incompatible changes between them, so we cannot upgrade to 1.19 for all versions, it should be 1.9 in older hadoop and 1.19 in hadoop3." This patch includes some copy and paste of the hadoop28 profile to a hadoop3 profile. Maven doesn't include anything like "profile inheritance". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3905) Change BUILDING.TXT to minimum JDK 1.8
Eric Wohlstadter created TEZ-3905: - Summary: Change BUILDING.TXT to minimum JDK 1.8 Key: TEZ-3905 URL: https://issues.apache.org/jira/browse/TEZ-3905 Project: Apache Tez Issue Type: Task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-3911) Optional mix/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr.
Eric Wohlstadter created TEZ-3911: - Summary: Optional mix/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr. Key: TEZ-3911 URL: https://issues.apache.org/jira/browse/TEZ-3911 Project: Apache Tez Issue Type: New Feature Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter Fix For: 0.9.next Consumers of HistoryLoggingService reported counters are currently required to compute any task-level aggregations other than "sum". This is inefficient as Tez is already "scanning" over this data. Computing incremental aggregates shouldn't require additional scans by ATS consumers. Provide an option for Task counter aggregations other than "sum". Computation of these extra counters can be turned on/off. The option will generate "synthetic" counters at final aggregation time for reporting to HistoryLoggingService, e.g. MAX_GC_TIME_MILLIS. Only incremental aggregations will be supported (min/max/avg). Aggregation computation will be folded into the existing "aggregation loop" beginning at VertexImpl.incrTaskCounters. Extra aggregations will only be supported during final counter aggregation. Aggregations will only include the "bestAttempt" for each task. A design doc will be provided. Because final task aggregation holds a lock, a performance report will be provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-4007) Zookeeper based FrameworkClient
Eric Wohlstadter created TEZ-4007: - Summary: Zookeeper based FrameworkClient Key: TEZ-4007 URL: https://issues.apache.org/jira/browse/TEZ-4007 Project: Apache Tez Issue Type: Sub-task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter A sub-class of FrameworkClient that interacts with application state managed in Zookeeper rather than Yarn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-4006) Make FrameworkClient Pluggable
Eric Wohlstadter created TEZ-4006:
-
Summary: Make FrameworkClient Pluggable
Key: TEZ-4006
URL: https://issues.apache.org/jira/browse/TEZ-4006
Project: Apache Tez
Issue Type: Sub-task
Reporter: Eric Wohlstadter
Assignee: Eric Wohlstadter
Currently {{FrameworkClient.createFrameworkClient}} is hardcoded to use either
LocalClient or YarnClient.
Change it to allow specifying a FrameworkClient impl. class from configuration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-4009) Zookeeper based AM Registry
Eric Wohlstadter created TEZ-4009: - Summary: Zookeeper based AM Registry Key: TEZ-4009 URL: https://issues.apache.org/jira/browse/TEZ-4009 Project: Apache Tez Issue Type: Sub-task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter Zookeeper based implementation of the interface defined by TEZ-4008. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-4008) Pluggable AM pool registry
Eric Wohlstadter created TEZ-4008:
-
Summary: Pluggable AM pool registry
Key: TEZ-4008
URL: https://issues.apache.org/jira/browse/TEZ-4008
Project: Apache Tez
Issue Type: Sub-task
Reporter: Eric Wohlstadter
Assignee: Eric Wohlstadter
Interface and reflective plugin configuration for a pool of AMs identified by a
namespace.
The registry should allow each {{DAGClientServer}} to register/unregister
themselves from a pool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-4014) Allow DAGAppMaster to read configuration from tez-site.xml plaintext
Eric Wohlstadter created TEZ-4014: - Summary: Allow DAGAppMaster to read configuration from tez-site.xml plaintext Key: TEZ-4014 URL: https://issues.apache.org/jira/browse/TEZ-4014 Project: Apache Tez Issue Type: Sub-task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter When launching DAGAppMaster independently, clients won't have to chance localize the configuration for DAGAppMaster in the protobuf format. In these cases it is more convenient for a platform to provide an AM configuration as a plaintext tez-site.xml. The change here will be to look for the protobuf conf first, and then fall back to looking for a tez-site.xml text file in the classpath. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-4016) Make TEZ_AM_LOCAL_RESOURCES_PB optional for unmanaged AM
Eric Wohlstadter created TEZ-4016:
-
Summary: Make TEZ_AM_LOCAL_RESOURCES_PB optional for unmanaged AM
Key: TEZ-4016
URL: https://issues.apache.org/jira/browse/TEZ-4016
Project: Apache Tez
Issue Type: Sub-task
Reporter: Eric Wohlstadter
Assignee: Eric Wohlstadter
In session mode, {{DAGAppMaster}} requires the
{{tez.session.local-resources.pb}} file to be present.
If resources are localized by a framework other than YARN, this may not be
required.
Make it optional.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-4024) DAGAppMaster should access resources in UGI context on startup
Eric Wohlstadter created TEZ-4024: - Summary: DAGAppMaster should access resources in UGI context on startup Key: TEZ-4024 URL: https://issues.apache.org/jira/browse/TEZ-4024 Project: Apache Tez Issue Type: Bug Affects Versions: 0.9.1 Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter On startup, DAGAppMaster accesses resources such as config PBs and staging dirs as yarn rather than through the UGI context. This creates access control violations when yarn does not have permissions for those resources. This also seems odd, since during shutdown, DAGAppMaster does access resources through UGI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-4009) Zookeeper based AM Registry
[ https://issues.apache.org/jira/browse/TEZ-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter resolved TEZ-4009. --- Resolution: Duplicate Subsumed by TEZ-4007 > Zookeeper based AM Registry > --- > > Key: TEZ-4009 > URL: https://issues.apache.org/jira/browse/TEZ-4009 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: TEZ-4009.1.patch > > > Zookeeper based implementation of the interface defined by TEZ-4008. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
