I modified the DMNetwork example to include the new DM with the modified
section. It has the same problems. Please find attached the code to this
email.

Thanks

On Tue, Feb 24, 2015 at 6:49 PM, Matthew Knepley <[email protected]> wrote:

> On Tue, Feb 24, 2015 at 6:42 PM, Miguel Angel Salazar de Troya <
> [email protected]> wrote:
>
>> I implemented the code as agreed, but I don't get the results I expected.
>> When I create the vector with DMCreateGlobalVector(), I obtain a vector
>> with a layout similar to the original DMNetwork, instead of the cloned
>> network with the new PetscSection. The code is as follows:
>>
>> DMClone(dm, &dmEdge);
>>
>> PetscSectionCreate(PETSC_COMM_WORLD, &s);
>> PetscSectionSetNumFields(s, 1);
>> PetscSectionSetFieldComponents(s, 0, 1);
>>
>> // Now to set the chart, I pick the edge range
>>
>> DMNetworkGetEdgeRange(dmEdge, & eStart, & eEnd)
>>
>> PetscSectionSetChart(s, eStart, eEnd);
>>
>> for(PetscInt e = eStart; c < eEnd; ++e) {
>>      PetscSectionSetDof(s, e, 1);
>>      PetscSectionSetFieldDof(s, e, 0, 1);
>> }
>> PetscSectionSetUp(s);
>>
>> DMSetDefaultSection(dmEdge s);
>> DMCreateGlobalVector(dmEdge, &globalVec);
>>
>> When I get into DMCreateGlobalVector(dmEdge, &globalVec) in the debugger,
>> in the function DMCreateSubDM_Section_Private() I call
>> PetscSectionView() on the section
>>
>
> I have no idea why you would be in DMCreateSubDM().
>
> Just view globalVec. If the code is as above, it will give you a vector
> with that layout. If not
> it should be trivial to make a small code and send it. I do this
> everywhere is PETSc, so the
> basic mechanism certainly works.
>
>   Thanks,
>
>      Matt
>
>
>> obtained by DMGetDefaultGlobalSection(dm, &sectionGlobal), and I obtain
>> a PetscSection nothing like the one I see when I call PetscSectionView()
>> on the PetscSection I created above. Does this have anything to do? I
>> tried to compare this strange PetscSection with the one from the original
>> DMNetwork, I call DMGetDefaultGlobalSection(dm, &sectionGlobal) before
>> the first line of the snippet above and I get this error message.
>>
>> 0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Object is in wrong state
>> [0]PETSC ERROR: DM must have a default PetscSection in order to create a
>> global PetscSection
>>
>> Thanks in advance
>> Miguel
>>
>>
>> On Mon, Feb 23, 2015 at 3:24 PM, Matthew Knepley <[email protected]>
>> wrote:
>>
>>> On Mon, Feb 23, 2015 at 2:15 PM, Miguel Angel Salazar de Troya <
>>> [email protected]> wrote:
>>>
>>>> Thanks a lot, the partition should be done before setting up the
>>>> section, right?
>>>>
>>>
>>> The partition will be automatic. All you have to do is make the local
>>> section. The DM is already partitioned,
>>> and the Section will inherit that.
>>>
>>>   Matt
>>>
>>>
>>>> Miguel
>>>>
>>>> On Mon, Feb 23, 2015 at 2:05 PM, Matthew Knepley <[email protected]>
>>>> wrote:
>>>>
>>>>> On Mon, Feb 23, 2015 at 1:40 PM, Miguel Angel Salazar de Troya <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Wouldn't including the edge variables in the global vector make the
>>>>>> code slower? I'm using the global vector in a TS, using one of the 
>>>>>> explicit
>>>>>> RK schemes. The edge variables would not be updated in the RHSFunction
>>>>>> evaluation. I only change the edge variables in the TSUpdate. If the 
>>>>>> global
>>>>>> vector had the edge variables, it would be a much larger vector, and all
>>>>>> the vector operations performed by the TS would be slower. Although the
>>>>>> vector F returned by the RHSFunction would be zero in the edge variable
>>>>>> components. I guess that being the vector sparse that would not be a
>>>>>> problem.
>>>>>>
>>>>>> I think I'm more interested in the PetscSection approach because it
>>>>>> might require less modifications in my code. However, I don't know how I
>>>>>> could do this. Maybe something like this?
>>>>>>
>>>>>> PetscSectionCreate(PETSC_COMM_WORLD, &s);
>>>>>> PetscSectionSetNumFields(s, 1);
>>>>>> PetscSectionSetFieldComponents(s, 0, 1);
>>>>>>
>>>>>> // Now to set the chart, I pick the edge range
>>>>>>
>>>>>> DMNetworkGetEdgeRange(dm, & eStart, & eEnd
>>>>>>
>>>>>> PetscSectionSetChart(s, eStart, eEnd);
>>>>>>
>>>>>> for(PetscInt e = eStart; c < eEnd; ++e) {
>>>>>>      PetscSectionSetDof(s, e, 1);
>>>>>>      PetscSectionSetFieldDof(s, e, 1, 1);
>>>>>>
>>>>>
>>>>> It should be PetscSectionSetFieldDof(s, e, 0, 1);
>>>>>
>>>>>
>>>>>> }
>>>>>> PetscSectionSetUp(s);
>>>>>>
>>>>>> Now in the manual I see this:
>>>>>>
>>>>>
>>>>> First you want to do:
>>>>>
>>>>>   DMClone(dm, &dmEdge);
>>>>>
>>>>> and then use dmEdge below.
>>>>>
>>>>>
>>>>>> DMSetDefaultSection(dm, s);
>>>>>> DMGetLocalVector(dm, &localVec);
>>>>>> DMGetGlobalVector(dm, &globalVec);
>>>>>>
>>>>>> Setting up the default section in the DM would interfere with the
>>>>>> section already set up with the variables in the vertices?
>>>>>>
>>>>>
>>>>> Yep, thats why you would use a clone.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> Thanks a lot for your responses.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 23, 2015 at 11:37 AM, Matthew Knepley <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> On Mon, Feb 23, 2015 at 9:27 AM, Miguel Angel Salazar de Troya <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> I'm iterating through local edges given in DMNetworkGetEdgeRange().
>>>>>>>> For each edge, I extract or modify its corresponding value in a global
>>>>>>>> petsc vector. Therefore that vector must have as many components as 
>>>>>>>> edges
>>>>>>>> there are in the network. To extract the value in the vector, I use
>>>>>>>> VecGetArray() and a variable counter that is incremented in each 
>>>>>>>> iteration.
>>>>>>>> The array that I obtain in VecGetArray() has to be the same size
>>>>>>>> than the edge range. That variable counter starts as 0, so if the array
>>>>>>>> that I obtained in VecGetArray() is x_array, x_array[0] must be
>>>>>>>> the component in the global vector that corresponds with the start edge
>>>>>>>> given in DMNetworkGetEdgeRange()
>>>>>>>>
>>>>>>>> I need that global petsc vector because I will use it in other
>>>>>>>> operations, it's not just data. Sorry for the confusion. Thanks in 
>>>>>>>> advance.
>>>>>>>>
>>>>>>>
>>>>>>> This sounds like an assembly operation. The usual paradigm is to
>>>>>>> compute in the local space, and then communicate to get to the global
>>>>>>> space. So you would make a PetscSection that had 1 (or some) unknowns on
>>>>>>> each cell (edge) and then you can use DMCreateGlobal/LocalVector() and
>>>>>>> DMLocalToGlobal() to do this.
>>>>>>>
>>>>>>> Does that make sense?
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>> Miguel
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 23, 2015 at 9:09 AM, Matthew Knepley <[email protected]
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> On Mon, Feb 23, 2015 at 8:42 AM, Miguel Angel Salazar de Troya <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks, that will help me. Now what I would like to have is the
>>>>>>>>>> following: if I have two processors and ten edges, the partitioning 
>>>>>>>>>> results
>>>>>>>>>> in the first processor having the edges 0-4 and the second 
>>>>>>>>>> processor, the
>>>>>>>>>> edges 5-9. I also have a global vector with as many components as 
>>>>>>>>>> edges,
>>>>>>>>>> 10. How can I partition it so the first processor also has the 0-4
>>>>>>>>>> components and the second, the 5-9 components of the vector?
>>>>>>>>>>
>>>>>>>>> I think it would help to know what you want to accomplish. This is
>>>>>>>>> how you are proposing to do it.'
>>>>>>>>>
>>>>>>>>> If you just want to put data on edges, DMNetwork has a facility
>>>>>>>>> for that already.
>>>>>>>>>
>>>>>>>>>   Thanks,
>>>>>>>>>
>>>>>>>>>      Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Miguel
>>>>>>>>>> On Feb 23, 2015 8:08 AM, "Abhyankar, Shrirang G." <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>>>  Miguel,
>>>>>>>>>>>    One possible way is to store the global numbering of any
>>>>>>>>>>> edge/vertex in the "component" attached to it. Once the mesh gets
>>>>>>>>>>> partitioned, the components are also distributed so you can easily 
>>>>>>>>>>> retrieve
>>>>>>>>>>> the global number of any edge/vertex by accessing its component. 
>>>>>>>>>>> This is
>>>>>>>>>>> what is done in the DMNetwork example pf.c although the global 
>>>>>>>>>>> numbering is
>>>>>>>>>>> not used for anything.
>>>>>>>>>>>
>>>>>>>>>>>  Shri
>>>>>>>>>>>  From: Matthew Knepley <[email protected]>
>>>>>>>>>>> Date: Mon, 23 Feb 2015 07:54:34 -0600
>>>>>>>>>>> To: Miguel Angel Salazar de Troya <[email protected]>
>>>>>>>>>>> Cc: "[email protected]" <[email protected]>
>>>>>>>>>>> Subject: Re: [petsc-users] DMNetworkGetEdgeRange() in parallel
>>>>>>>>>>>
>>>>>>>>>>>   On Sun, Feb 22, 2015 at 3:59 PM, Miguel Angel Salazar de
>>>>>>>>>>> Troya <[email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks. Once I obtain that Index Set with the routine 
>>>>>>>>>>>> DMPlexCreateCellNumbering()
>>>>>>>>>>>> (I assume that the edges in DMNetwork correspond to cells in 
>>>>>>>>>>>> DMPlex) can I
>>>>>>>>>>>> use it to partition a vector with as many components as edges I 
>>>>>>>>>>>> have in my
>>>>>>>>>>>> network?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  I do not completely understand the question.
>>>>>>>>>>>
>>>>>>>>>>>  If you want a partition of the edges, you can use
>>>>>>>>>>> DMPlexCreatePartition() and its friend DMPlexDistribute(). What
>>>>>>>>>>> are you trying to do?
>>>>>>>>>>>
>>>>>>>>>>>     Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>  Thanks
>>>>>>>>>>>> Miguel
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Feb 22, 2015 at 12:15 PM, Matthew Knepley <
>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>  On Sun, Feb 22, 2015 at 11:01 AM, Miguel Angel Salazar de
>>>>>>>>>>>>> Troya <[email protected]> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  I noticed that the routine DMNetworkGetEdgeRange() returns
>>>>>>>>>>>>>> the local indices for the edge range. Is there any way to obtain 
>>>>>>>>>>>>>> the global
>>>>>>>>>>>>>> indices? So if my network has 10 edges, the processor 1 has the 
>>>>>>>>>>>>>> 0-4 edges
>>>>>>>>>>>>>> and the processor 2, the 5-9 edges, how can I obtain this 
>>>>>>>>>>>>>> information?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>  One of the points of DMPlex is we do not require a global
>>>>>>>>>>>>> numbering. Everything is numbered
>>>>>>>>>>>>> locally, and the PetscSF maps local numbers to local numbers
>>>>>>>>>>>>> in order to determine ownership.
>>>>>>>>>>>>>
>>>>>>>>>>>>>  If you want to create a global numbering for some reason,
>>>>>>>>>>>>> you can using DMPlexCreatePointNumbering().
>>>>>>>>>>>>> There are also cell and vertex versions that we use for
>>>>>>>>>>>>> output, so you could do it just for edges as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>>    Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>>       Matt
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>  Thanks
>>>>>>>>>>>>>>  Miguel
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  --
>>>>>>>>>>>>>>  *Miguel Angel Salazar de Troya*
>>>>>>>>>>>>>> Graduate Research Assistant
>>>>>>>>>>>>>> Department of Mechanical Science and Engineering
>>>>>>>>>>>>>> University of Illinois at Urbana-Champaign
>>>>>>>>>>>>>> (217) 550-2360
>>>>>>>>>>>>>> [email protected]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>  --
>>>>>>>>>>>>> What most experimenters take for granted before they begin
>>>>>>>>>>>>> their experiments is infinitely more interesting than any results 
>>>>>>>>>>>>> to which
>>>>>>>>>>>>> their experiments lead.
>>>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>  --
>>>>>>>>>>>>  *Miguel Angel Salazar de Troya*
>>>>>>>>>>>> Graduate Research Assistant
>>>>>>>>>>>> Department of Mechanical Science and Engineering
>>>>>>>>>>>> University of Illinois at Urbana-Champaign
>>>>>>>>>>>> (217) 550-2360
>>>>>>>>>>>> [email protected]
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to 
>>>>>>>>>>> which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which 
>>>>>>>>> their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Miguel Angel Salazar de Troya*
>>>>>>>> Graduate Research Assistant
>>>>>>>> Department of Mechanical Science and Engineering
>>>>>>>> University of Illinois at Urbana-Champaign
>>>>>>>> (217) 550-2360
>>>>>>>> [email protected]
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which 
>>>>>>> their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Miguel Angel Salazar de Troya*
>>>>>> Graduate Research Assistant
>>>>>> Department of Mechanical Science and Engineering
>>>>>> University of Illinois at Urbana-Champaign
>>>>>> (217) 550-2360
>>>>>> [email protected]
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Miguel Angel Salazar de Troya*
>>>> Graduate Research Assistant
>>>> Department of Mechanical Science and Engineering
>>>> University of Illinois at Urbana-Champaign
>>>> (217) 550-2360
>>>> [email protected]
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>>
>> --
>> *Miguel Angel Salazar de Troya*
>> Graduate Research Assistant
>> Department of Mechanical Science and Engineering
>> University of Illinois at Urbana-Champaign
>> (217) 550-2360
>> [email protected]
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
*Miguel Angel Salazar de Troya*
Graduate Research Assistant
Department of Mechanical Science and Engineering
University of Illinois at Urbana-Champaign
(217) 550-2360
[email protected]
function mpc = case9
% CASE9    Power flow data for 9 bus, 3 generator case.
% MATPOWER case9 data
% http://www.pserc.cornell.edu//matpower/

% R. D. Zimmerman, C. E. Murillo-Sánchez, and R. J. Thomas, 
% "MATPOWER: Steady-State Operations, Planning and Analysis Tools 
% for Power Systems Research and Education," Power Systems, IEEE 
% Transactions on, vol. 26, no. 1, pp. 12-19, Feb. 2011

%%-----  Power Flow Data  -----%%
%% system MVA base
mpc.baseMVA = 100;

%% bus data
%	bus_i	type	Pd	Qd	Gs	Bs	area	Vm	Va	baseKV	zone	Vmax	Vmin
mpc.bus = [
	1	3	0	0	0	0	1	1	0	345	1	1.1	0.9;
	2	2	0	0	0	0	1	1	0	345	1	1.1	0.9;
	3	2	0	0	0	0	1	1	0	345	1	1.1	0.9;
	4	1	0	0	0	0	1	1	0	345	1	1.1	0.9;
	5	1	90	30	0	0	1	1	0	345	1	1.1	0.9;
	6	1	0	0	0	0	1	1	0	345	1	1.1	0.9;
	7	1	100	35	0	0	1	1	0	345	1	1.1	0.9;
	8	1	0	0	0	0	1	1	0	345	1	1.1	0.9;
	9	1	125	50	0	0	1	1	0	345	1	1.1	0.9;
];

%% generator data
%	bus	Pg	Qg	Qmax	Qmin	Vg	mBase	status	Pmax	Pmin	Pc1	Pc2	Qc1min	Qc1max	Qc2min	Qc2max	ramp_agc	ramp_10	ramp_30	ramp_q	apf
mpc.gen = [
	1	0	0	300	-300	1	100	1	250	10	0	0	0	0	0	0	0	0	0	0	0;
	2	163	0	300	-300	1	100	1	300	10	0	0	0	0	0	0	0	0	0	0	0;
	3	85	0	300	-300	1	100	1	270	10	0	0	0	0	0	0	0	0	0	0	0;
];

%% branch data
%	fbus	tbus	r	x	b	rateA	rateB	rateC	ratio	angle	status	angmin	angmax
mpc.branch = [
	1	4	0	0.0576	0	250	250	250	0	0	1	-360	360;
	4	5	0.017	0.092	0.158	250	250	250	0	0	1	-360	360;
	5	6	0.039	0.17	0.358	150	150	150	0	0	1	-360	360;
	3	6	0	0.0586	0	300	300	300	0	0	1	-360	360;
	6	7	0.0119	0.1008	0.209	150	150	150	0	0	1	-360	360;
	7	8	0.0085	0.072	0.149	250	250	250	0	0	1	-360	360;
	8	2	0	0.0625	0	250	250	250	0	0	1	-360	360;
	8	9	0.032	0.161	0.306	250	250	250	0	0	1	-360	360;
	9	4	0.01	0.085	0.176	250	250	250	0	0	1	-360	360;
];

%%-----  OPF Data  -----%%
%% area data
%	area	refbus
mpc.areas = [
	1	5;
];

%% generator cost data
%	1	startup	shutdown	n	x1	y1	...	xn	yn
%	2	startup	shutdown	n	c(n-1)	...	c0
mpc.gencost = [
	2	1500	0	3	0.11	5	150;
	2	2000	0	3	0.085	1.2	600;
	2	3000	0	3	0.1225	1	335;
];

Attachment: makefile
Description: Binary data

static char help[] = "This example demonstrates the use of DMNetwork interface for solving a nonlinear electric power grid problem.\n\
                      The available solver options are in the pfoptions file and the data files are in the datafiles directory.\n\
                      The data file format used is from the MatPower package (http://www.pserc.cornell.edu//matpower/).\n\
                      Run this program: mpiexec -n <n> ./PF\n\
                      mpiexec -n <n> ./PF -pfdata <filename>\n";

/* T
   Concepts: DMNetwork
   Concepts: PETSc SNES solver
*/

#include "pf.h"
#include <petscdmnetwork.h>

PetscMPIInt rank;

#undef __FUNCT__
#define __FUNCT__ "GetListofEdges"
PetscErrorCode GetListofEdges(PetscInt nbranches, EDGEDATA branch,int edges[])
{
  PetscInt       i, fbus,tbus;

  PetscFunctionBegin;
  
  for (i=0; i < nbranches; i++) {
    fbus = branch[i].internal_i;
    tbus = branch[i].internal_j;
    edges[2*i]   = fbus;
    edges[2*i+1] = tbus;
  }
  PetscFunctionReturn(0);
}

typedef struct{
  PetscScalar  Sbase;
}UserCtx;

#undef __FUNCT__
#define __FUNCT__ "FormFunction"
PetscErrorCode FormFunction(SNES snes,Vec X, Vec F,void *appctx)
{
  PetscErrorCode ierr;
  DM             networkdm;
  UserCtx       *User=(UserCtx*)appctx;
  Vec           localX,localF;
  PetscInt      e;
  PetscInt      v,vStart,vEnd,vfrom,vto;
  const PetscScalar *xarr;
  PetscScalar   *farr;
  PetscInt      offsetfrom,offsetto,offset;
  DMNetworkComponentGenericDataType *arr;

  PetscFunctionBegin;
  ierr = SNESGetDM(snes,&networkdm);CHKERRQ(ierr);
  ierr = DMGetLocalVector(networkdm,&localX);CHKERRQ(ierr);
  ierr = DMGetLocalVector(networkdm,&localF);CHKERRQ(ierr);
  ierr = VecSet(F,0.0);CHKERRQ(ierr);

  ierr = DMGlobalToLocalBegin(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr);
  ierr = DMGlobalToLocalEnd(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr);

  ierr = DMGlobalToLocalBegin(networkdm,F,INSERT_VALUES,localF);CHKERRQ(ierr);
  ierr = DMGlobalToLocalEnd(networkdm,F,INSERT_VALUES,localF);CHKERRQ(ierr);

  ierr = VecGetArrayRead(localX,&xarr);CHKERRQ(ierr);
  ierr = VecGetArray(localF,&farr);CHKERRQ(ierr);

  ierr = DMNetworkGetVertexRange(networkdm,&vStart,&vEnd);CHKERRQ(ierr);
  ierr = DMNetworkGetComponentDataArray(networkdm,&arr);CHKERRQ(ierr);

  for (v=vStart; v < vEnd; v++) {
    PetscInt    i,j,offsetd,key;
    PetscScalar Vm;
    PetscScalar Sbase=User->Sbase;
    VERTEXDATA  bus;
    GEN         gen;
    LOAD        load;
    PetscBool   ghostvtex;
    PetscInt    numComps;

    ierr = DMNetworkIsGhostVertex(networkdm,v,&ghostvtex);CHKERRQ(ierr);
    ierr = DMNetworkGetNumComponents(networkdm,v,&numComps);CHKERRQ(ierr);
    ierr = DMNetworkGetVariableOffset(networkdm,v,&offset);CHKERRQ(ierr);
    for (j = 0; j < numComps; j++) {
      ierr = DMNetworkGetComponentTypeOffset(networkdm,v,j,&key,&offsetd);CHKERRQ(ierr);
      if (key == 1) {
	bus = (VERTEXDATA)(arr+offsetd);
	/* Handle reference bus constrained dofs */
	if (bus->ide == REF_BUS || bus->ide == ISOLATED_BUS) {
	  farr[offset] = xarr[offset] - bus->va*PETSC_PI/180.0;
	  farr[offset+1] = xarr[offset+1] - bus->vm;
	  break;
	}

	if (!ghostvtex) {
	  Vm = xarr[offset+1];

	  /* Shunt injections */
	  farr[offset] += Vm*Vm*bus->gl/Sbase;
	  if(bus->ide != PV_BUS) farr[offset+1] += -Vm*Vm*bus->bl/Sbase;
	}
	PetscInt nconnedges;
	const PetscInt *connedges;

	ierr = DMNetworkGetSupportingEdges(networkdm,v,&nconnedges,&connedges);CHKERRQ(ierr);
	for (i=0; i < nconnedges; i++) {
	  EDGEDATA branch;
	  PetscInt keye;
	  e = connedges[i];
	  ierr = DMNetworkGetComponentTypeOffset(networkdm,e,0,&keye,&offsetd);CHKERRQ(ierr);
	  branch = (EDGEDATA)(arr+offsetd);
	  if (!branch->status) continue;
	  PetscScalar Gff,Bff,Gft,Bft,Gtf,Btf,Gtt,Btt;
	  Gff = branch->yff[0];
	  Bff = branch->yff[1];
	  Gft = branch->yft[0];
	  Bft = branch->yft[1];
	  Gtf = branch->ytf[0];
	  Btf = branch->ytf[1];
	  Gtt = branch->ytt[0];
	  Btt = branch->ytt[1];

	  const PetscInt *cone;
	  ierr = DMNetworkGetConnectedNodes(networkdm,e,&cone);CHKERRQ(ierr);
	  vfrom = cone[0];
	  vto   = cone[1];

	  ierr = DMNetworkGetVariableOffset(networkdm,vfrom,&offsetfrom);CHKERRQ(ierr);
	  ierr = DMNetworkGetVariableOffset(networkdm,vto,&offsetto);CHKERRQ(ierr);

	  PetscScalar Vmf,Vmt,thetaf,thetat,thetaft,thetatf;

	  thetaf = xarr[offsetfrom];
	  Vmf     = xarr[offsetfrom+1];
	  thetat = xarr[offsetto];
	  Vmt     = xarr[offsetto+1];
	  thetaft = thetaf - thetat;
	  thetatf = thetat - thetaf;

	  if (vfrom == v) {
	    farr[offsetfrom]   += Gff*Vmf*Vmf + Vmf*Vmt*(Gft*cos(thetaft) + Bft*sin(thetaft));
	    farr[offsetfrom+1] += -Bff*Vmf*Vmf + Vmf*Vmt*(-Bft*cos(thetaft) + Gft*sin(thetaft));
	  } else {
	    farr[offsetto]   += Gtt*Vmt*Vmt + Vmt*Vmf*(Gtf*cos(thetatf) + Btf*sin(thetatf));
	    farr[offsetto+1] += -Btt*Vmt*Vmt + Vmt*Vmf*(-Btf*cos(thetatf) + Gtf*sin(thetatf));
	  }
	}
      } else if (key == 2) {
	if (!ghostvtex) {
	  gen = (GEN)(arr+offsetd);
	  if (!gen->status) continue;
	  farr[offset] += -gen->pg/Sbase;
	  farr[offset+1] += -gen->qg/Sbase;
	}
      } else if (key == 3) {
	if (!ghostvtex) {
	  load = (LOAD)(arr+offsetd);
	  farr[offset] += load->pl/Sbase;
	  farr[offset+1] += load->ql/Sbase;
	}
      }
    }
    if (bus->ide == PV_BUS) {
      farr[offset+1] = xarr[offset+1] - bus->vm;
    }
  }
  ierr = VecRestoreArrayRead(localX,&xarr);CHKERRQ(ierr);
  ierr = VecRestoreArray(localF,&farr);CHKERRQ(ierr);
  ierr = DMRestoreLocalVector(networkdm,&localX);CHKERRQ(ierr);

  ierr = DMLocalToGlobalBegin(networkdm,localF,ADD_VALUES,F);CHKERRQ(ierr);
  ierr = DMLocalToGlobalEnd(networkdm,localF,ADD_VALUES,F);CHKERRQ(ierr);
  ierr = DMRestoreLocalVector(networkdm,&localF);CHKERRQ(ierr);

  PetscFunctionReturn(0);
}

#undef __FUNCT__
#define __FUNCT__ "FormJacobian"
PetscErrorCode FormJacobian(SNES snes,Vec X, Mat J,Mat Jpre,void *appctx)
{
  PetscErrorCode ierr;
  DM            networkdm;
  UserCtx       *User=(UserCtx*)appctx;
  Vec           localX;
  PetscInt      e;
  PetscInt      v,vStart,vEnd,vfrom,vto;
  const PetscScalar *xarr;
  PetscInt      offsetfrom,offsetto,goffsetfrom,goffsetto;
  DMNetworkComponentGenericDataType *arr;
  PetscInt      row[2],col[8];
  PetscScalar   values[8];

  PetscFunctionBegin;
  ierr = MatZeroEntries(J);CHKERRQ(ierr);

  ierr = SNESGetDM(snes,&networkdm);CHKERRQ(ierr);
  ierr = DMGetLocalVector(networkdm,&localX);CHKERRQ(ierr);

  ierr = DMGlobalToLocalBegin(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr);
  ierr = DMGlobalToLocalEnd(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr);

  ierr = VecGetArrayRead(localX,&xarr);CHKERRQ(ierr);

  ierr = DMNetworkGetVertexRange(networkdm,&vStart,&vEnd);CHKERRQ(ierr);
  ierr = DMNetworkGetComponentDataArray(networkdm,&arr);CHKERRQ(ierr);

  for (v=vStart; v < vEnd; v++) {
    PetscInt    i,j,offsetd,key;
    PetscInt    offset,goffset;
    PetscScalar Vm;
    PetscScalar Sbase=User->Sbase;
    VERTEXDATA  bus;
    PetscBool   ghostvtex;
    PetscInt    numComps;

    ierr = DMNetworkIsGhostVertex(networkdm,v,&ghostvtex);CHKERRQ(ierr);
    ierr = DMNetworkGetNumComponents(networkdm,v,&numComps);CHKERRQ(ierr);
    for (j = 0; j < numComps; j++) {
      ierr = DMNetworkGetVariableOffset(networkdm,v,&offset);CHKERRQ(ierr);
      ierr = DMNetworkGetVariableGlobalOffset(networkdm,v,&goffset);CHKERRQ(ierr);
      ierr = DMNetworkGetComponentTypeOffset(networkdm,v,j,&key,&offsetd);CHKERRQ(ierr);
      if (key == 1) {
	bus = (VERTEXDATA)(arr+offsetd);
	if (!ghostvtex) {
	  /* Handle reference bus constrained dofs */
	  if (bus->ide == REF_BUS || bus->ide == ISOLATED_BUS) {
	    row[0] = goffset; row[1] = goffset+1;
	    col[0] = goffset; col[1] = goffset+1; col[2] = goffset; col[3] = goffset+1;
	    values[0] = 1.0; values[1] = 0.0; values[2] = 0.0; values[3] = 1.0;
	    ierr = MatSetValues(J,2,row,2,col,values,ADD_VALUES);CHKERRQ(ierr);
	    break;
	  }
	  
	  Vm = xarr[offset+1];
	  
	  /* Shunt injections */
          row[0] = goffset; row[1] = goffset+1;
          col[0] = goffset; col[1] = goffset+1;
          values[0] = values[1] = values[2] = values[3] = 0.0;
          if(bus->ide != PV_BUS) {
            values[1] = 2*Vm*bus->gl/Sbase;
            values[3] = -2*Vm*bus->bl/Sbase;
          }
          ierr = MatSetValues(J,2,row,2,col,values,ADD_VALUES);CHKERRQ(ierr);
	}

	PetscInt nconnedges;
	const PetscInt *connedges;

	ierr = DMNetworkGetSupportingEdges(networkdm,v,&nconnedges,&connedges);CHKERRQ(ierr);
	for (i=0; i < nconnedges; i++) {
	  EDGEDATA branch;
	  VERTEXDATA busf,bust;
	  PetscInt   offsetfd,offsettd,keyf,keyt;
	  e = connedges[i];
	  ierr = DMNetworkGetComponentTypeOffset(networkdm,e,0,&key,&offsetd);CHKERRQ(ierr);
	  branch = (EDGEDATA)(arr+offsetd);
	  if (!branch->status) continue;
	  PetscScalar Gff,Bff,Gft,Bft,Gtf,Btf,Gtt,Btt;
	  Gff = branch->yff[0];
	  Bff = branch->yff[1];
	  Gft = branch->yft[0];
	  Bft = branch->yft[1];
	  Gtf = branch->ytf[0];
	  Btf = branch->ytf[1];
	  Gtt = branch->ytt[0];
	  Btt = branch->ytt[1];

	  const PetscInt *cone;
	  ierr = DMNetworkGetConnectedNodes(networkdm,e,&cone);CHKERRQ(ierr);
	  vfrom = cone[0];
	  vto   = cone[1];

	  ierr = DMNetworkGetVariableOffset(networkdm,vfrom,&offsetfrom);CHKERRQ(ierr);
	  ierr = DMNetworkGetVariableOffset(networkdm,vto,&offsetto);CHKERRQ(ierr);
	  ierr = DMNetworkGetVariableGlobalOffset(networkdm,vfrom,&goffsetfrom);CHKERRQ(ierr);
	  ierr = DMNetworkGetVariableGlobalOffset(networkdm,vto,&goffsetto);CHKERRQ(ierr);

	  if (goffsetfrom < 0) goffsetfrom = -goffsetfrom - 1; /* Convert to actual global offset for ghost nodes, global offset is -(gstart+1) */
	  if (goffsetto < 0) goffsetto = -goffsetto - 1;
	  PetscScalar Vmf,Vmt,thetaf,thetat,thetaft,thetatf;

	  thetaf = xarr[offsetfrom];
	  Vmf     = xarr[offsetfrom+1];
	  thetat = xarr[offsetto];
	  Vmt     = xarr[offsetto+1];
	  thetaft = thetaf - thetat;
	  thetatf = thetat - thetaf;

	  ierr = DMNetworkGetComponentTypeOffset(networkdm,vfrom,0,&keyf,&offsetfd);CHKERRQ(ierr);
	  ierr = DMNetworkGetComponentTypeOffset(networkdm,vto,0,&keyt,&offsettd);CHKERRQ(ierr);
	  busf = (VERTEXDATA)(arr+offsetfd);
	  bust = (VERTEXDATA)(arr+offsettd);

	  if (vfrom == v) {
	    if (busf->ide != REF_BUS) {
	      /*    farr[offsetfrom]   += Gff*Vmf*Vmf + Vmf*Vmt*(Gft*cos(thetaft) + Bft*sin(thetaft));  */
	      row[0]  = goffsetfrom;
	      col[0]  = goffsetfrom; col[1] = goffsetfrom+1; col[2] = goffsetto; col[3] = goffsetto+1;
	      values[0] =  Vmf*Vmt*(Gft*-sin(thetaft) + Bft*cos(thetaft)); /* df_dthetaf */    
	      values[1] =  2*Gff*Vmf + Vmt*(Gft*cos(thetaft) + Bft*sin(thetaft)); /* df_dVmf */
	      values[2] =  Vmf*Vmt*(Gft*sin(thetaft) + Bft*-cos(thetaft)); /* df_dthetat */
	      values[3] =  Vmf*(Gft*cos(thetaft) + Bft*sin(thetaft)); /* df_dVmt */
	      
	      ierr = MatSetValues(J,1,row,4,col,values,ADD_VALUES);CHKERRQ(ierr);
	    }
	    if (busf->ide != PV_BUS && busf->ide != REF_BUS) {
	      row[0] = goffsetfrom+1;
	      col[0]  = goffsetfrom; col[1] = goffsetfrom+1; col[2] = goffsetto; col[3] = goffsetto+1;
	      /*    farr[offsetfrom+1] += -Bff*Vmf*Vmf + Vmf*Vmt*(-Bft*cos(thetaft) + Gft*sin(thetaft)); */
	      values[0] =  Vmf*Vmt*(Bft*sin(thetaft) + Gft*cos(thetaft));
	      values[1] =  -2*Bff*Vmf + Vmt*(-Bft*cos(thetaft) + Gft*sin(thetaft));
	      values[2] =  Vmf*Vmt*(-Bft*sin(thetaft) + Gft*-cos(thetaft));
	      values[3] =  Vmf*(-Bft*cos(thetaft) + Gft*sin(thetaft));
	      
	      ierr = MatSetValues(J,1,row,4,col,values,ADD_VALUES);CHKERRQ(ierr);
	    }
	  } else {
	    if(bust->ide != REF_BUS) {
	      row[0] = goffsetto;
	      col[0] = goffsetto; col[1] = goffsetto+1; col[2] = goffsetfrom; col[3] = goffsetfrom+1;
	      /*    farr[offsetto]   += Gtt*Vmt*Vmt + Vmt*Vmf*(Gtf*cos(thetatf) + Btf*sin(thetatf)); */
	      values[0] =  Vmt*Vmf*(Gtf*-sin(thetatf) + Btf*cos(thetaft)); /* df_dthetat */
	      values[1] =  2*Gtt*Vmt + Vmf*(Gtf*cos(thetatf) + Btf*sin(thetatf)); /* df_dVmt */
	      values[2] =  Vmt*Vmf*(Gtf*sin(thetatf) + Btf*-cos(thetatf)); /* df_dthetaf */
	      values[3] =  Vmt*(Gtf*cos(thetatf) + Btf*sin(thetatf)); /* df_dVmf */
	      
	      ierr = MatSetValues(J,1,row,4,col,values,ADD_VALUES);CHKERRQ(ierr);
	    }
	    if (bust->ide != PV_BUS && bust->ide != REF_BUS) {
	      row[0] = goffsetto+1;
	      col[0] = goffsetto; col[1] = goffsetto+1; col[2] = goffsetfrom; col[3] = goffsetfrom+1;
	      /*    farr[offsetto+1] += -Btt*Vmt*Vmt + Vmt*Vmf*(-Btf*cos(thetatf) + Gtf*sin(thetatf)); */
	      values[0] =  Vmt*Vmf*(Btf*sin(thetatf) + Gtf*cos(thetatf));
	      values[1] =  -2*Btt*Vmt + Vmf*(-Btf*cos(thetatf) + Gtf*sin(thetatf));
	      values[2] =  Vmt*Vmf*(-Btf*sin(thetatf) + Gtf*-cos(thetatf));
	      values[3] =  Vmt*(-Btf*cos(thetatf) + Gtf*sin(thetatf));
	      
	      ierr = MatSetValues(J,1,row,4,col,values,ADD_VALUES);CHKERRQ(ierr);
	    }
	  }
	}
	if (!ghostvtex && bus->ide == PV_BUS) {
	  row[0] = goffset+1; col[0] = goffset+1;
	  values[0]  = 1.0;
	  ierr = MatSetValues(J,1,row,1,col,values,ADD_VALUES);CHKERRQ(ierr);
	}
      }
    }
  }
  ierr = VecRestoreArrayRead(localX,&xarr);CHKERRQ(ierr);
  ierr = DMRestoreLocalVector(networkdm,&localX);CHKERRQ(ierr);

  ierr = MatAssemblyBegin(J,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  ierr = MatAssemblyEnd(J,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);

  PetscFunctionReturn(0);
}

#undef __FUNCT__
#define __FUNCT__ "SetInitialValues"
PetscErrorCode SetInitialValues(DM networkdm,Vec X,void* appctx)
{
  PetscErrorCode ierr;
  VERTEXDATA     bus;
  GEN            gen;
  PetscInt       v, vStart, vEnd, offset;
  PetscBool      ghostvtex;
  Vec            localX;
  PetscScalar    *xarr;
  PetscInt       key;
  DMNetworkComponentGenericDataType *arr;
  
  PetscFunctionBegin;
  ierr = DMNetworkGetVertexRange(networkdm,&vStart, &vEnd);CHKERRQ(ierr);

  ierr = DMGetLocalVector(networkdm,&localX);CHKERRQ(ierr);

  ierr = VecSet(X,0.0);CHKERRQ(ierr);
  ierr = DMGlobalToLocalBegin(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr);
  ierr = DMGlobalToLocalEnd(networkdm,X,INSERT_VALUES,localX);CHKERRQ(ierr);

  ierr = VecGetArray(localX,&xarr);CHKERRQ(ierr);
  ierr = DMNetworkGetComponentDataArray(networkdm,&arr);CHKERRQ(ierr);
  for (v = vStart; v < vEnd; v++) {
    ierr = DMNetworkIsGhostVertex(networkdm,v,&ghostvtex);CHKERRQ(ierr);
    if (ghostvtex) continue;
    PetscInt numComps;
    PetscInt j;
    PetscInt offsetd;
    ierr = DMNetworkGetVariableOffset(networkdm,v,&offset);CHKERRQ(ierr);
    ierr = DMNetworkGetNumComponents(networkdm,v,&numComps);CHKERRQ(ierr);
    for (j=0; j < numComps; j++) {
      ierr = DMNetworkGetComponentTypeOffset(networkdm,v,j,&key,&offsetd);CHKERRQ(ierr);
      if (key == 1) {
	bus = (VERTEXDATA)(arr+offsetd);
	xarr[offset] = bus->va*PETSC_PI/180.0;
	xarr[offset+1] = bus->vm;
      } else if(key == 2) {
	gen = (GEN)(arr+offsetd);
	if (!gen->status) continue;
	xarr[offset+1] = gen->vs; 
	break;
      }
    }
  }
  ierr = VecRestoreArray(localX,&xarr);CHKERRQ(ierr);
  ierr = DMLocalToGlobalBegin(networkdm,localX,ADD_VALUES,X);CHKERRQ(ierr);
  ierr = DMLocalToGlobalEnd(networkdm,localX,ADD_VALUES,X);CHKERRQ(ierr);
  ierr = DMRestoreLocalVector(networkdm,&localX);CHKERRQ(ierr);
  PetscFunctionReturn(0);
}


#undef __FUNCT__
#define __FUNCT__ "main"
int main(int argc,char ** argv)
{
  PetscErrorCode       ierr;
  char                 pfdata_file[PETSC_MAX_PATH_LEN]="case9.m";
  PFDATA               pfdata;
  PetscInt             numEdges=0,numVertices=0;
  int                  *edges = NULL;
  PetscInt             i;  
  DM                   networkdm;
  PetscInt             componentkey[4];
  UserCtx              User;
  PetscLogStage        stage1,stage2;
  PetscInt             size;

  PetscInitialize(&argc,&argv,"pfoptions",help);

  ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr);

  /* Create an empty network object */
  ierr = DMNetworkCreate(PETSC_COMM_WORLD,&networkdm);CHKERRQ(ierr);
  /* Register the components in the network */
  ierr = DMNetworkRegisterComponent(networkdm,"branchstruct",sizeof(struct _p_EDGEDATA),&componentkey[0]);CHKERRQ(ierr);
  ierr = DMNetworkRegisterComponent(networkdm,"busstruct",sizeof(struct _p_VERTEXDATA),&componentkey[1]);CHKERRQ(ierr);
  ierr = DMNetworkRegisterComponent(networkdm,"genstruct",sizeof(struct _p_GEN),&componentkey[2]);CHKERRQ(ierr);
  ierr = DMNetworkRegisterComponent(networkdm,"loadstruct",sizeof(struct _p_LOAD),&componentkey[3]);CHKERRQ(ierr);

  ierr = PetscLogStageRegister("Read Data",&stage1);CHKERRQ(ierr);
  PetscLogStagePush(stage1);
  /* READ THE DATA */
  if (!rank) {
    /*    READ DATA */
    /* Only rank 0 reads the data */
    ierr = PetscOptionsGetString(PETSC_NULL,"-pfdata",pfdata_file,PETSC_MAX_PATH_LEN-1,NULL);CHKERRQ(ierr);
    ierr = PFReadMatPowerData(&pfdata,pfdata_file);CHKERRQ(ierr);
    User.Sbase = pfdata.sbase;

    numEdges = pfdata.nbranch;
    numVertices = pfdata.nbus;

    ierr = PetscMalloc(2*numEdges*sizeof(int),&edges);CHKERRQ(ierr);
    ierr = GetListofEdges(pfdata.nbranch,pfdata.branch,edges);CHKERRQ(ierr);
  }
  PetscLogStagePop();
  MPI_Barrier(PETSC_COMM_WORLD);
  ierr = PetscLogStageRegister("Create network",&stage2);CHKERRQ(ierr);
  PetscLogStagePush(stage2);
  /* Set number of nodes/edges */
  ierr = DMNetworkSetSizes(networkdm,numVertices,numEdges,PETSC_DETERMINE,PETSC_DETERMINE);CHKERRQ(ierr);
  /* Add edge connectivity */
  ierr = DMNetworkSetEdgeList(networkdm,edges);CHKERRQ(ierr);
  /* Set up the network layout */
  ierr = DMNetworkLayoutSetUp(networkdm);CHKERRQ(ierr);

  if (!rank) {
    ierr = PetscFree(edges);CHKERRQ(ierr);
  }
  /* Add network components */
  PetscInt eStart, eEnd, vStart, vEnd,j;
  PetscInt genj=0,loadj=0;
  ierr = DMNetworkGetEdgeRange(networkdm,&eStart,&eEnd);CHKERRQ(ierr);
  for (i = eStart; i < eEnd; i++) {
    ierr = DMNetworkAddComponent(networkdm,i,componentkey[0],&pfdata.branch[i-eStart]);CHKERRQ(ierr);
  }
  ierr = DMNetworkGetVertexRange(networkdm,&vStart,&vEnd);CHKERRQ(ierr);
  for (i = vStart; i < vEnd; i++) {
    ierr = DMNetworkAddComponent(networkdm,i,componentkey[1],&pfdata.bus[i-vStart]);CHKERRQ(ierr);
    if (pfdata.bus[i-vStart].ngen) {
      for (j = 0; j < pfdata.bus[i-vStart].ngen; j++) {
	ierr = DMNetworkAddComponent(networkdm,i,componentkey[2],&pfdata.gen[genj++]);CHKERRQ(ierr);
      }
    }
    if (pfdata.bus[i-vStart].nload) {
      for (j=0; j < pfdata.bus[i-vStart].nload; j++) {
	ierr = DMNetworkAddComponent(networkdm,i,componentkey[3],&pfdata.load[loadj++]);CHKERRQ(ierr);
      }
    }
    /* Add number of variables */
    ierr = DMNetworkAddNumVariables(networkdm,i,2);CHKERRQ(ierr);
  }
  /* Set up DM for use */
  ierr = DMSetUp(networkdm);CHKERRQ(ierr);

  if (!rank) {
    ierr = PetscFree(pfdata.bus);CHKERRQ(ierr);
    ierr = PetscFree(pfdata.gen);CHKERRQ(ierr);
    ierr = PetscFree(pfdata.branch);CHKERRQ(ierr);
    ierr = PetscFree(pfdata.load);CHKERRQ(ierr);
  }


  ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr);
  if (size > 1) {
    DM distnetworkdm;
    /* Network partitioning and distribution of data */
    ierr = DMNetworkDistribute(networkdm,0,&distnetworkdm);CHKERRQ(ierr);
    ierr = DMDestroy(&networkdm);CHKERRQ(ierr);
    networkdm = distnetworkdm;
  }

  PetscLogStagePop();
  ierr = DMNetworkGetEdgeRange(networkdm,&eStart,&eEnd);CHKERRQ(ierr);
  ierr = DMNetworkGetVertexRange(networkdm,&vStart,&vEnd);CHKERRQ(ierr);

  /*
   * Building the PetscSection for the edges
   */

  DM edgesdm;
  PetscSection edgesSection;


  ierr =  DMClone(networkdm,&edgesdm);CHKERRQ(ierr);
  ierr =  PetscSectionCreate(PETSC_COMM_WORLD, &edgesSection);CHKERRQ(ierr);
  ierr =  PetscSectionSetNumFields(edgesSection, 1);CHKERRQ(ierr);
  ierr =  PetscSectionSetFieldComponents(edgesSection, 0, 1);CHKERRQ(ierr);

  // Now to set the chart, I pick the edge range
  ierr =  DMNetworkGetEdgeRange(edgesdm, & eStart, & eEnd);CHKERRQ(ierr);
  ierr =  PetscSectionSetChart(edgesSection, eStart, eEnd);CHKERRQ(ierr);

  for(i = eStart; i < eEnd; ++i) {
	   ierr = PetscSectionSetDof(edgesSection, i, 1);CHKERRQ(ierr);
	   ierr =  PetscSectionSetFieldDof(edgesSection, i, 0, 1);CHKERRQ(ierr);
  }
  ierr =  PetscSectionSetUp(edgesSection);CHKERRQ(ierr);
  ierr =  DMSetDefaultSection(edgesdm, edgesSection);CHKERRQ(ierr);

  /* Set up DM for use */
  ierr = DMSetUp(edgesdm);CHKERRQ(ierr);

  PetscSectionView(edgesSection, PETSC_VIEWER_STDOUT_WORLD);

  /*
   * Obtain global vector
   */
  Vec globalVec;
  ierr =  DMCreateGlobalVector(edgesdm, &globalVec);CHKERRQ(ierr);




 PetscSynchronizedPrintf(PETSC_COMM_WORLD,"eStart = %d\n", eStart);
 PetscSynchronizedPrintf(PETSC_COMM_WORLD,"eEnd = %d\n", eEnd);

 PetscSynchronizedPrintf(PETSC_COMM_WORLD,"vStart = %d\n", vStart);
 PetscSynchronizedPrintf(PETSC_COMM_WORLD,"vEnd = %d\n", vEnd);
  
#if 0
  PetscInt numComponents;
  EDGEDATA edge;
  PetscInt offset,key;
  DMNetworkComponentGenericDataType *arr;
  for (i = eStart; i < eEnd; i++) {
    ierr = DMNetworkGetComponentDataArray(networkdm,&arr);CHKERRQ(ierr);
    ierr = DMNetworkGetComponentTypeOffset(networkdm,i,0,&key,&offset);CHKERRQ(ierr);
    edge = (EDGEDATA)(arr+offset);
    ierr = DMNetworkGetNumComponents(networkdm,i,&numComponents);CHKERRQ(ierr);
    ierr = PetscPrintf(PETSC_COMM_SELF,"Rank %d ncomps = %d Line %d ---- %d\n",rank,numComponents,edge->internal_i,edge->internal_j);CHKERRQ(ierr);
  }    

  VERTEXDATA bus;
  GEN        gen;
  LOAD       load;
  PetscInt   kk;
  for (i = vStart; i < vEnd; i++) {
    ierr = DMNetworkGetComponentDataArray(networkdm,&arr);CHKERRQ(ierr);
    ierr = DMNetworkGetNumComponents(networkdm,i,&numComponents);CHKERRQ(ierr);
    for (kk=0; kk < numComponents; kk++) {
      ierr = DMNetworkGetComponentTypeOffset(networkdm,i,kk,&key,&offset);CHKERRQ(ierr);
      if (key == 1) {
	bus = (VERTEXDATA)(arr+offset);
	ierr = PetscPrintf(PETSC_COMM_SELF,"Rank %d ncomps = %d Bus %d\n",rank,numComponents,bus->internal_i);CHKERRQ(ierr);
      } else if (key == 2) {
	gen = (GEN)(arr+offset);
	ierr = PetscPrintf(PETSC_COMM_SELF,"Rank %d Gen pg = %f qg = %f\n",rank,gen->pg,gen->qg);CHKERRQ(ierr);
      } else if (key == 3) {
	load = (LOAD)(arr+offset);
	ierr = PetscPrintf(PETSC_COMM_SELF,"Rank %d Load pd = %f qd = %f\n",rank,load->pl,load->ql);CHKERRQ(ierr);
      }
    }
  }  
#endif
  /* Broadcast Sbase to all processors */
  ierr = MPI_Bcast(&User.Sbase,1,MPI_DOUBLE,0,PETSC_COMM_WORLD);CHKERRQ(ierr);

  Vec X,F;
  ierr = DMCreateGlobalVector(networkdm,&X);CHKERRQ(ierr);
  ierr = VecDuplicate(X,&F);CHKERRQ(ierr);

  /*
   * Compare both vectors
   */
  ierr = PetscPrintf(PETSC_COMM_SELF,"Edge Vector\n");CHKERRQ(ierr);
  VecView(globalVec,PETSC_VIEWER_STDOUT_WORLD);

  ierr = PetscPrintf(PETSC_COMM_SELF,"Network vector \n");CHKERRQ(ierr);
  VecView(X,PETSC_VIEWER_STDOUT_WORLD);


  Mat J;
  ierr = DMCreateMatrix(networkdm,&J);CHKERRQ(ierr);
  ierr = MatSetOption(J,MAT_NEW_NONZERO_ALLOCATION_ERR,PETSC_FALSE);CHKERRQ(ierr);

  ierr = SetInitialValues(networkdm,X,&User);CHKERRQ(ierr);

  SNES snes;
  /* HOOK UP SOLVER */
  ierr = SNESCreate(PETSC_COMM_WORLD,&snes);CHKERRQ(ierr);
  ierr = SNESSetDM(snes,networkdm);CHKERRQ(ierr);
  ierr = SNESSetFunction(snes,F,FormFunction,&User);CHKERRQ(ierr);
  ierr = SNESSetJacobian(snes,J,J,FormJacobian,&User);CHKERRQ(ierr);
  ierr = SNESSetFromOptions(snes);CHKERRQ(ierr);

  ierr = SNESSolve(snes,NULL,X);CHKERRQ(ierr);

  ierr = VecDestroy(&X);CHKERRQ(ierr);
  ierr = VecDestroy(&F);CHKERRQ(ierr);
  ierr = MatDestroy(&J);CHKERRQ(ierr);

  ierr = SNESDestroy(&snes);CHKERRQ(ierr);
  ierr = DMDestroy(&networkdm);CHKERRQ(ierr);

  PetscFinalize();
  return 0;
}
#ifndef PF_H
#define PF_H

# include "petsc.h"

# define MAXLINE 1000
#define REF_BUS 3
#define PV_BUS 2
#define PQ_BUS 1
#define ISOLATED_BUS 4
#define NGEN_AT_BUS_MAX 15
#define NLOAD_AT_BUS_MAX 1

/* 2. Bus data */
/* 11 columns */
struct _p_VERTEXDATA{
  PetscInt      bus_i; /* Integer bus number .. used by some formats like Matpower */
  char	 	i[20]; /* Bus Number */
  char 		name[20]; /* Bus Name */
  PetscScalar 	basekV; /* Bus Base kV */
  PetscInt 	ide; /* Bus type code */
  PetscScalar 	gl; /* Active component of shunt admittance to ground */
  PetscScalar 	bl; /* Reactive component of shunt admittance to ground */
  PetscInt 	area; /* Area number */
  PetscInt 	zone; /* Zone number */
  PetscScalar 	vm; /* Bus voltage magnitude; in pu */
  PetscScalar 	va; /* Bus voltage phase angle */
  PetscInt 	owner; /* Owner number */
  PetscInt	internal_i; /* Internal Bus Number */
  PetscInt      ngen; /* Number of generators incident at this bus */
  PetscInt      gidx[NGEN_AT_BUS_MAX]; /* list of inndices for accessing the generator data in GEN structure */
  PetscInt      nload;
  PetscInt      lidx[NLOAD_AT_BUS_MAX];
};

typedef struct _p_VERTEXDATA *VERTEXDATA;

/* 3. Load data */
/* 12 columns */
struct _p_LOAD{
  PetscInt      bus_i; /* Bus number */
  char 		i[20]; /* Bus Number or extended bus name*/
  char 		id[20]; /* Load identifier, in case of multiple loads. 1 by default */
  PetscInt 	status; /* Load status */
  PetscInt 	area; /* Area to which load is assigned */
  PetscInt 	zone; /* Zone to which load is assigned */
  PetscScalar 	pl; /* Active power component of constant MVA load */
  PetscScalar 	ql; /* Reactive power component of constant MVA load */
  PetscScalar 	ip; /* Active power component of constant current load: MW pu V */
  PetscScalar 	iq; /* Reactive power component of constant current load: Mvar pu V */
  PetscScalar 	yp; /* Active power component of constant admittance load: MW pu V */
  PetscScalar 	yq; /* Reactive power component of constant admittance load: Mvar pu V */
  PetscInt 	owner; /* Owner number */
  PetscInt	internal_i; /* Internal Bus Number */
  PetscScalar   scale_load;
};

typedef struct _p_LOAD *LOAD;

/* 4. Generator data */
/* 20+ columns */
/******* 20, USING ONLY 1 OWNER's WORTH OF DATA. COME BACK TO THIS LATER, if necessary ******/
struct _p_GEN{
  PetscInt      bus_i;
  char 		i[20]; /* Bus Number or extended bus name*/
  char 		id[20]; /* Generator identifier, in case of multiple generators at same bus. 1 by default */
  PetscScalar 	pg; /* Generator active power output */
  PetscScalar 	qg; /* Generator reactive power output */
  PetscScalar 	qt; /* Maximum reactive power output: Mvar */
  PetscScalar 	qb; /* Minimum reactive power output: Mvar */
  PetscScalar 	vs; /* Regulated voltage setpoint: pu */
  PetscInt 	ireg; /* Remote bus number/identifier */
  PetscScalar 	mbase; /* MVA base of the machine */
  PetscScalar 	zr; /* Complex machine impedance ZSOURCE in pu on mbase */
  PetscScalar 	zx; /* ----------------------"------------------------- */
  PetscScalar 	rt; /* Step-up transformer impedance XTRAN in pu on mbase */
  PetscScalar 	xt; /* -----------------------"-------------------------- */
  PetscScalar 	gtap; /* Step-up transformer turns ratio */
  PetscInt 	status; /* Machine status */
  PetscScalar 	rmpct; /* Mvar % required to hold voltage at remote bus */
  PetscScalar 	pt; /* Gen max active power output: MW */
  PetscScalar 	pb; /* Gen min active power output: MW */
  PetscInt 	o1; /* Owner number */
  PetscScalar 	f1; /* Fraction of ownership */
  PetscInt	internal_i; /* Internal Bus Number */
  PetscScalar   scale_gen;
};

typedef struct _p_GEN *GEN;

/* 17+ columns */
struct _p_EDGEDATA{
  PetscInt      fbus;
  PetscInt      tbus;
  char 		i[20]; /* Bus Number or extended bus name*/
  char 		j[20]; /* Bus Number or extended bus name*/
  char 		ckt[20]; /* Circuit identifier. 1 by default */
  PetscScalar 	r; /* Branch resistance: pu */
  PetscScalar 	x; /* Branch reactance: pu */
  PetscScalar 	b; /* Branch charging susceptance: pu */
  PetscScalar 	rateA; /* rate A in MVA */
  PetscScalar 	rateB; /* rate B in MVA */
  PetscScalar 	rateC; /* rate C in MVA */
  PetscScalar   tapratio;
  PetscScalar   phaseshift;
  PetscScalar 	gi; /* Complex admittance at 'i' end: pu */
  PetscScalar 	bi; /* Complex admittance at 'i' end: pu */
  PetscScalar 	gj; /* Complex admittance at 'j' end: pu */
  PetscScalar 	bj; /* Complex admittance at 'j' end: pu */
  PetscInt 	status; /* Service status */
  PetscScalar 	length; /* Line length */
  PetscInt 	o1; /* Owner number */
  PetscScalar 	f1; /* Fraction of ownership */
  PetscInt	internal_i; /* Internal From Bus Number */
  PetscInt	internal_j; /* Internal To Bus Number */
  PetscScalar   yff[2],yft[2],ytf[2],ytt[2]; /* [G,B] */
};

typedef struct _p_EDGEDATA *EDGEDATA;

/* PTI format data structure */
typedef struct{
  PetscScalar sbase; /* System base MVA */
  PetscInt    nbus,ngen,nbranch,nload; /* # of buses,gens,branches, and loads (includes elements which are
                                          out of service */
  VERTEXDATA bus;
  LOAD load;
  GEN gen;
  EDGEDATA branch;
}PFDATA;

extern PetscErrorCode PFReadMatPowerData(PFDATA*,char*);
#endif

Attachment: pfoptions
Description: Binary data

#include "petscmat.h"
#include "pf.h"
#include <string.h>
#include <ctype.h>
  
#undef __FUNCT__
#define __FUNCT__ "PFReadMatPowerData"
PetscErrorCode PFReadMatPowerData(PFDATA *pf,char *filename)
{
  FILE           *fp;
  PetscErrorCode ierr;
  VERTEXDATA     Bus;
  LOAD           Load;
  GEN            Gen;
  EDGEDATA       Branch;
  PetscInt       line_counter=0;
  PetscInt       bus_start_line=-1,bus_end_line=-1; /* xx_end_line points to the next line after the record ends */
  PetscInt       gen_start_line=-1,gen_end_line=-1;
  PetscInt       br_start_line=-1,br_end_line=-1;
  char           line[MAXLINE];
  PetscInt       loadi=0,geni=0,bri=0,busi=0,i;
  PetscInt       extbusnum,bustype_i,j;
  PetscScalar    Pd,Qd;
  PetscInt       maxbusnum=-1,intbusnum;

  PetscFunctionBegin;

  fp = fopen(filename,"r");
  /* Check for valid file */
  if (fp == NULL)  {
     ierr = PetscPrintf(PETSC_COMM_WORLD,"Can't open Matpower data file\n");CHKERRQ(ierr);
     exit(EXIT_FAILURE);
  }
  pf->nload=0;
  while(fgets(line,MAXLINE,fp) != NULL) {
    if(strstr(line,"mpc.bus") != NULL)    bus_start_line = line_counter+1; /* Bus data starts from next line */
    if(strstr(line,"mpc.gen") != NULL && gen_start_line == -1)    gen_start_line = line_counter+1; /* Generator data starts from next line */
    if(strstr(line,"mpc.branch") != NULL) br_start_line = line_counter+1; /* Branch data starts from next line */
    if(strstr(line,"];") != NULL) {
      if (bus_start_line != -1 && bus_end_line == -1) bus_end_line = line_counter;
      if (gen_start_line != -1 && gen_end_line == -1) gen_end_line = line_counter;
      if (br_start_line  != -1 && br_end_line == -1) br_end_line = line_counter;
    }

    /* Count the number of pq loads */
    if(bus_start_line != -1 && line_counter >= bus_start_line && bus_end_line == -1) {
      sscanf(line,"%d %d %lf %lf",&extbusnum,&bustype_i,&Pd,&Qd);
      if(!((Pd == 0.0) && (Qd == 0.0))) pf->nload++;
      if (extbusnum > maxbusnum) maxbusnum = extbusnum;
    }
    line_counter++;
  }
  fclose(fp);

  pf->nbus    = bus_end_line - bus_start_line;
  pf->ngen    = gen_end_line - gen_start_line;
  pf->nbranch = br_end_line  - br_start_line;

  ierr = PetscPrintf(PETSC_COMM_SELF,"nb = %d, ngen = %d, nload = %d, nbranch = %d\n",pf->nbus,pf->ngen,pf->nload,pf->nbranch);CHKERRQ(ierr);

  ierr = PetscMalloc(pf->nbus*sizeof(struct _p_VERTEXDATA),&pf->bus);CHKERRQ(ierr);
  ierr = PetscMalloc(pf->ngen*sizeof(struct _p_GEN),&pf->gen);CHKERRQ(ierr);
  ierr = PetscMalloc(pf->nload*sizeof(struct _p_LOAD),&pf->load);CHKERRQ(ierr);
  ierr = PetscMalloc(pf->nbranch*sizeof(struct _p_EDGEDATA),&pf->branch);CHKERRQ(ierr);
  Bus = pf->bus; Gen = pf->gen; Load = pf->load; Branch = pf->branch;

  for(i=0; i < pf->nbus; i++) {
    pf->bus[i].ngen = pf->bus[i].nload = 0;
  }

  /* Setting pf->sbase to 100 */
  pf->sbase = 100.0;

  PetscInt *busext2intmap;
  ierr = PetscMalloc1(maxbusnum+1,&busext2intmap);CHKERRQ(ierr);
  for(i=0; i < maxbusnum+1; i++) busext2intmap[i] = -1;

  fp = fopen(filename,"r");
  /* Reading data */
  for(i=0;i<line_counter;i++) {
    fgets(line,MAXLINE,fp);

    if((i >= bus_start_line) && (i < bus_end_line)) {
      /* Bus data */
      sscanf(line,"%d %d %lf %lf %lf %lf %d %lf %lf %lf",		\
	     &Bus[busi].bus_i,&Bus[busi].ide,&Pd,&Qd,&Bus[busi].gl,	\
	     &Bus[busi].bl,&Bus[busi].area,&Bus[busi].vm,&Bus[busi].va,&Bus[busi].basekV);
      Bus[busi].internal_i = busi;
      busext2intmap[Bus[busi].bus_i] = busi;

      if(!((Pd == 0.0) && (Qd == 0.0))) {
	Load[loadi].bus_i = Bus[busi].bus_i;
	Load[loadi].status = 1;
	Load[loadi].pl = Pd;
	Load[loadi].ql = Qd;
	Load[loadi].area = Bus[busi].area;
	Load[loadi].internal_i = busi;
	Bus[busi].lidx[Bus[busi].nload++] = loadi;
	if (Bus[busi].nload > NLOAD_AT_BUS_MAX)
	  SETERRQ(PETSC_COMM_SELF,PETSC_ERR_SUP,"Exceeded maximum number of loads allowed at bus");
	loadi++;
      }
      busi++;
    }

    /* Read generator data */
    if(i >= gen_start_line && i < gen_end_line) {
      sscanf(line,"%d %lf %lf %lf %lf %lf %lf %d %lf %lf",&Gen[geni].bus_i, \
	     &Gen[geni].pg,&Gen[geni].qg,&Gen[geni].qt,&Gen[geni].qb, \
	     &Gen[geni].vs,&Gen[geni].mbase,&Gen[geni].status,&Gen[geni].pt, \
	     &Gen[geni].pb);

      intbusnum = busext2intmap[Gen[geni].bus_i];
      Gen[geni].internal_i = intbusnum;
      Bus[intbusnum].gidx[Bus[intbusnum].ngen++] = geni;

      Bus[intbusnum].vm = Gen[geni].vs;

      if (Bus[intbusnum].ngen > NGEN_AT_BUS_MAX) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_SUP,"Exceeded maximum number of generators allowed at bus");
      geni++;
    }
    
    if(i >= br_start_line && i < br_end_line) {
      sscanf(line,"%d %d %lf %lf %lf %lf %lf %lf %lf %lf %d",&Branch[bri].fbus,&Branch[bri].tbus, \
	     &Branch[bri].r,&Branch[bri].x,&Branch[bri].b,		\
	     &Branch[bri].rateA,&Branch[bri].rateB,&Branch[bri].rateC, \
	     &Branch[bri].tapratio,&Branch[bri].phaseshift,&Branch[bri].status);
      if(!Branch[bri].tapratio) Branch[bri].tapratio = 1.0;
      Branch[bri].phaseshift *= PETSC_PI/180.0;

      intbusnum = busext2intmap[Branch[bri].fbus];
      Branch[bri].internal_i = intbusnum;

      intbusnum = busext2intmap[Branch[bri].tbus];
      Branch[bri].internal_j = intbusnum;

      /* Compute self and transfer admittances */
      PetscScalar R,X,Bc,B,G,Zm,tap,shift,tap2,tapr,tapi;
      R = Branch[bri].r;
      X = Branch[bri].x;
      Bc = Branch[bri].b;

      Zm = R*R + X*X;
      G  = R/Zm;
      B  = -X/Zm;

      tap = Branch[bri].tapratio;
      shift = Branch[bri].phaseshift;
      tap2 = tap*tap;
      tapr = tap*cos(shift);
      tapi = tap*sin(shift);

      Branch[bri].yff[0] = G/tap2; 
      Branch[bri].yff[1] = (B+Bc/2.0)/tap2;
      
      Branch[bri].yft[0] = -(G*tapr - B*tapi)/tap2;
      Branch[bri].yft[1] = -(B*tapr + G*tapi)/tap2;

      Branch[bri].ytf[0] = -(G*tapr + B*tapi)/tap2;
      Branch[bri].ytf[1] = -(B*tapr - G*tapi)/tap2;

      Branch[bri].ytt[0] = G;
      Branch[bri].ytt[1] = B+Bc/2.0;

      bri++;
    }
  }
  fclose(fp);
  
  /* Reorder the generator data structure according to bus numbers */
  GEN  newgen;
  LOAD newload;
  PetscInt genj=0,loadj=0;
  ierr = PetscMalloc(pf->ngen*sizeof(struct _p_GEN),&newgen);CHKERRQ(ierr);
  ierr = PetscMalloc(pf->nload*sizeof(struct _p_LOAD),&newload);CHKERRQ(ierr);
  for (i = 0; i < pf->nbus; i++) {
    for (j = 0; j < pf->bus[i].ngen; j++) {
      ierr = PetscMemcpy(&newgen[genj++],&pf->gen[pf->bus[i].gidx[j]],sizeof(struct _p_GEN));
    }
    for (j = 0; j < pf->bus[i].nload; j++) {
      ierr = PetscMemcpy(&newload[loadj++],&pf->load[pf->bus[i].lidx[j]],sizeof(struct _p_LOAD));
    }
  }
  ierr = PetscFree(pf->gen);CHKERRQ(ierr);
  ierr = PetscFree(pf->load);CHKERRQ(ierr);
  pf->gen = newgen;
  pf->load = newload;

  ierr = PetscFree(busext2intmap);CHKERRQ(ierr);
  PetscFunctionReturn(0);
}

Reply via email to